ASIAN DEVELOPMENT BANK

SST: 98010

SPECIAL EVALUATION STUDY OF

FACTORS AFFECTING PROJECT PERFORMANCE

IN THE AGRICULTURE AND SOCIAL SECTORS:

A REVIEW OF POSTEVALUATION REPORTS

BETWEEN 1991 AND 1997

December 1998

ABBREVIATIONS

AGSO AGSS APEP ARDE AWOD BRM BTOR COSO COSS DMC EA EDAN EIRR ENIN GDP HAP IES IRAD MTSF NRM PAC PCR PEIS PEO PEWD PGIR PPAR PPTA REIRR RRP TCR TPAR URD WB WSS

Agriculture and Social Infrastructure Agricultural and Support Services Annual Performance Evaluation Program Annual Review of Development Effectiveness Office of the Director Agriculture and Social Sectors Department (West) Bangladesh Resident Mission Back-to-Office Report Central Operations Services Office Country Operational Strategy Study developing member country Executing Agency Economic Analysis and Research Division Economic Internal Rate of Return Energy and Infrastructure gross domestic product Health and Population Impact Evaluation Study Irrigation and Rural Development Medium-Term Strategic Framework Nepal Resident Mission Project Administration Committee Project Completion Reports Postevaluation Information System Postevaluation Office Evaluation Division West percent gap in EIRR Project Performance Audit Report Project Preparatory Technical Assistance Reestimated EIRR at Postevaluation Report and Recommendation of the President Technical Assistance Completion Report Technical Assistance Performance Audit Report urban development and housing World Bank Water Supply and Sanitation

Postevaluation Office, SS 31

CONTENTS EXECUTIVE SUMMARY I. INTRODUCTION A. B. C. D. II. A. B. 1. 2. 3. Background and Rationale Objectives and Scope of Activities Limitations of the Study Finalizing the Report ii 1 1 6 7 8 8 8 10 10 10 11 12 12 13 15 17 19 20

MAJOR FINDINGS OF THE STUDY Performance Measure Typically Monitored Indicators of Project Performance Delays Cost Underrun/Overrun Bank Supervision

C. Indicators of Project Performance Specific to AGSO Sectors 1. 2. 3. 4. 5. III. The Banks Performance Performance of the Consultants During Implementation Performance of the Executing Agency Performance of the Borrower External Factors CONCLUSIONS AND IMPLICATIONS

EXECUTIVE SUMMARY Historical trends of overall project performance, by year of approval and year of postevaluation, revealed that the performance of projects in the agriculture and social infrastructure (AGSO) sectors was consistently lower than that in the energy and infrastructure (ENIN) sectors (Figures 1-4). The study attempts to clarify the major causes of project success and failure that are specific to AGSO and ENIN sector groups, and draw some implications and suggestions useful in addressing issues pertaining to the performance of projects in the AGSO sectors. This is a pilot attempt to use statistical procedures in assessing the performance of Bank-assisted projects. The study is based mainly on information readily available in 141 project performance audit reports on projects approved between 1981 and 1992 and postevaluated between 1991 and 1997. Projects approved after 1992, when the Bank adopted the second Medium-Term Strategic Framework, are not yet ready for postevaluation. The study involved statistical analysis using regression models. Two sets of multivariate regression models were built. The models were based on a conceptual framework comprising the major participants in a project. The first set of models focused on identifying the distinction between the typical factors that affect project performance in AGSO and ENIN sector groups. The second set assessed the significance of typical as well as other factors specific to AGSO sectors. The models used two project performance measures: the reestimated economic internal rates of return (REIRR) at postevaluation, and the deviation of REIRR from the EIRR estimate at appraisal, measured by the percent gap in EIRR. The study was limited by data and information constraints. In the regression models, only EIRRs could be easily incorporated as dependent variables; consequently, the models included only 34 projects from AGSO sectors, of which only 3 were social infrastructure. Information constraints made it difficult to analyze the impact of such variables as participatory approach and environmentall of which are highly relevant under the Banks current policies as projects referring to them were limited. Caution was exercised in applying the results of the statistical tests that confirm and quantify the relationships between the measures of project performance and the causative factors. The tests were conducted for only a limited number of cases. The findings summarized here were concurred by operations departments (especially those concerned with AGSO sectors) based on their experience. The study found that managing time and cost overruns alone was insufficient to improve the performance of projects in AGSO sectors. Such factors significantly affected project performance only in ENIN sectors. The study indicated that, in the case of projects in AGSO sectors, attention should be paid as well to other factorsboth project specific and external whose impact on project performance is large. This suggests a weakness in the Banks traditional project administration mechanism where disproportionate attention had been directed to managing time and cost. The study underscores the importance of the newly introduced project performance reportwhich comprehensively reviews the various aspects of a project

iii compared with the traditional Project Administration Committee notes, especially for social infrastructure projects. The models identified the following significant determinants of project performance in AGSO sectors (Figures 5-6). At the country level, the macroeconomic environment is the dominant factor affecting the REIRR. The borrowers performance is also a critical determinant of project success. Good borrower performance implies a sector policy environment favorable to project operation, governments commitment to the project, and provision of domestic funds. Concerning borrower performance, project success in AGSO sectors depends on the Executing Agencys capability to supervise project implementation and properly utilize facilities. Statistical results show that adequate consulting services during implementation also yield significant rewards, by sharply reducing the gap between the appraisal and postevaluation EIRRs. The Banks foremost contribution can come at the project preparatory technical assistance (PPTA) stage where, in particular, greater realism is expected. The study revealed that the PPTA was weak for projects in AGSO sectors (Figures 7-8). The implications of the findings for Bank policy and future operations are highlighted in the following paragraphs. At the country level, the Bank should more realistically assess the macroeconomic and policy environment, and choose a set of interventions that best fit, given the existing constraints. At the project level, in countries where the policy framework is distorted and the implementation environment is weak, necessary advisory and capacity building TA services should in many cases precede lending activities. These services should not be attached to loans. At the fact-finding stage of the PPTA, and subsequently at project appraisal, the Bank should carefully assess the capacity of the Executing Agency for project implementation, utilization of facilities, and commitment to maintenance (adequacy of staff skills and funds). The Banks continuing focus on encouraging commitment to ownership of project goals by increasing Executing Agency involvement in design and implementation is well underscored. Important lessons were learned from the study and valuable comments and suggestions received from operations departments are as follows: Timely and relevant cognizance of social issues as critical determinants of project success should be analyzed. A more detailed treatment of exogenous and endogenous variables should be carried out so as to launch remedial measures for improving project performance. A closer review is needed to ascertain why most successful AGSO projects were

iv prepared using a PPTA, while many AGSO projects other than those rated as generally successful were only partly successful or unsuccessful despite having had a PPTA.

I.

INTRODUCTION

A.

Background and Rationale

1. In 1992, the Bank set five strategic development objectives.1 The targeted project mix for public sector projects was set at 50:50: at least 50 percent of projects should have social or environmental objectives either as a primary or secondary objective. The Bank's MediumTerm Strategic Framework (1995-1998) confirmed the continued relevance of these objectives and the project mix, and redefined the mission of the Bank from financing development projects to providing an integrated package of development services.2 2. In line with such a strategic operational thrust, 194 (or 58 percent) of 333 projects approved during the past five years had social or environmental improvement as a primary or secondary objective. Thirty-two percent of these are agricultural projects, while 51 percent are in social infrastructure, together reaching 83 percent. In addition, irrespective of their having social and environmental primary or secondary objectives, projects in the agriculture and social infrastructure (AGSO) sectors are meant to bring about more extensive indirect social and environmental advancement than projects in other sectors. Therefore, achievement of the Banks strategic development objectives heavily relies on the performance of Bank projects in AGSO sectors. 3. However, a cursory review of the postevaluated projects presents some worrisome features concerning the past performance of projects in AGSO sectors (Box 1). Of the 257 postevaluated projects3 in these sectors that were approved between 1968 and 1992, only 44 percent were rated as generally successful, while 81 percent of projects in the energy and infrastructure (ENIN) sectors were rated as generally successful. The rating of unsuccessful was given to 15 percent of projects in AGSO sectors and to only 4 percent in ENIN sectors (Table 1).
Box 1 The Postevaluation Offices (PEO's) evaluation reports and other studies assess the causes of project success and failure, and extract lessons learned as feedback to ongoing Bank operations. Annual reviews of postevaluation findings reveal a number of causes of project failure, some of which are recurrent. Furthermore, clear differences in the performance of projects in AGSO and ENIN sectors suggest that sectoral elements affect performance. Statistical analysis of a sample of evaluated projects could produce information useful in identifying possible generic and structural constraints inherent in socioeconomic conditions of developing member countries (DMCs) and in Bank operations. The findings from this analysis could enable the Bank to address such constraints through a sector or strategic approach.

2 3

The objectives are (i) promoting economic growth, (ii) reducing poverty, (iii) supporting human development (including population planning), (iv) improving the status of women, and (v) protecting the environment. These include support for policy reforms, capacity building, and regional cooperation. 257 projects (or 50 percent) belong to AGSO sectors, while 185 projects (or 36 percent) fall in ENIN sectors, with the remaining 73 projects belonging to other sectors.

3
Table 1: Assessment of Overall Performance of Projects
a

Sectors

Rating

All Projects
b

Number AGSO All Generally Successful Partly Successful Unsuccessful All Generally Successful Partly Successful Unsuccessful All 257 112 106 38 185 150 27 8 73 515

Percent 50 44 41 15 36 81 15 4 14 100

Projects Approved Between 1981 and 1992 and Postevaluated Between 1991 and 1997 b Number Percent 88 34 43 10 39 32 7 0 14 141 62 39 49 11 28 82 18 0 10 100

ENIN

Others Total
a b

No rating was provided for one project in AGSO sector. Percentages for All ratings are calculated with respect to the total number of projects, while percentages for individual rating categories (generally successful, partly successful, and unsuccessful) are calculated with respect to the number of projects under All ratings in each sector group.

4. A closer look at the projects approved between 1981 and 1992 and postevaluated between 1991 and 1997 reveals that the gap between the two groups of sectors is considerable: only 39 percent of the projects in AGSO sectors were rated as generally successful as against 82 percent for projects in ENIN sectors. 5. Tables 2 and 3 contrast the historical trends of overall project performance, by year of postevaluation and year of approval, respectively. Projects in AGSO sectors consistently performed less satisfactorily than those in ENIN sectors in each evaluation year (Table 2). Table 3 confirms the same for each approval year.1 While all of these projects, except one, were processed prior to 1992 and before the Bank adopted Medium-Term Strategic Framework and a strong emphasis on social and environmental objectives, the findings provide a critical hint on the future achievement of the Banks strategic development objectives as well as the overall quality of the Banks project portfolioparticularly if sufficient measures have not been taken to redress the situation. It is generally acknowledged that the developmental challenge is higher in AGSO projects due to their greater complexity, weaker institutions, and other factors including the large number of direct beneficiaries.

Analysis of the worsening trend in the performance of AGSO projects in relation to ENIN projects is beyond the scope of the study, but could be taken up in a future study.

4
Table 2: Overall Performance of Projects, by Evaluation Year Projects (Number) by Evaluation Year 1992 1993 1994 1995 1996 3 5 7 2 1 8 4 5 0 2 6 5 6 1 1 3 4 7 1 0 4 4 7 1 1

Rating Generally Successful Partly Successful Unsuccessful

a

Sectors AGSO ENIN AGSO ENIN AGSO

1991 6 4 5 2 1

1997 4 6 6 0 4

Percentage of projects that were rated as Generally Successful All sectors 52 44 57 AGSO 50 27 50 ENIN 67 71 100

60 46 83

47 30 80

45 33 80

45 29 100

a b

No project in ENIN sectors was rated unsuccessful. Percentages for All sectors are with respect to the total number of projects in all sectors. Percentages for AGSO and ENIN sector groups are with respect to the total number of projects in All ratings (including No rating).

Table 3: Overall Performance of Projects, by Year of Approval Projects (Number) by Year of Approval 1984 1985 1986 1987 1988 1989 1990 4 6 3 0 2 6 3 7 0 0 2 1 4 1 0 0 6 1 1 1 2 6 2 1 1 0 0 2 0 1 1 0 1 1 0

Rating Sectors 1981 1982 1983 GS PS US AGSO ENIN AGSO ENIN AGSO 6 1 9 2 2 7 2 7 0 1 6 5 7 1 2

1991 0 1 0 0 0

1992 0 1 0 0 0

Percentage of projects that were rated as GSa All sectors 35 47 52 67 50 AGSO 35 47 40 44 46 ENIN 33 100 83 100 100

50 33 50

60 0 86

62 40 86

29 na na

25 na na

100 na na

100 na na

na = not available, GS = generally successful, PS = partly successful, US = unsuccessful, AGSO = agriculture and social infrastructure, ENIN = energy and infrastructure.

5
a

Percentages for All sectors are with respect to the total number of projects in all sectors. Percentages for AGSO and ENIN sector groups are with respect to the total number of projects in All ratings (including No rating).

6 B. Objectives and Scope of Activities

6. The ultimate objective of the study is to improve understanding of factors affecting the performance of Bank-financed projects. 7. (i) (ii) (iii) In line with this objective, the study attempts to answer the following questions: What are the major factors systematically affecting project performance? Are there such factors specific to AGSO and ENIN sector groups? How significantly does each factor affect the performance/rating of projects in AGSO sectors? What are the implications of the study findings for the Bank and the DMCs especially in terms of Bank policies and future operations? The scope included (i) review of project performance audit reports (PPARs) on projects approved from 1981 to 1992 and postevaluated between 1991 and 1997 (Appendix 1, Table 1), other relevant postevaluation reports, and other reports of the Bank and the World Bank (WB) (Appendix 2); and creation of a database of the information, including relevant additional information from the Postevaluation Information System (PEIS) (Appendix 3); statistical analysis (Box 2) of typical factors relating to cost, design, implementation, operation, borrower performance, etc. as judged in the PPARs, to identify sector elements of the factors specific to AGSO and ENIN sectors; and

(iv)

8.

(ii)

(iii) implications regarding future actions to address strategically the generic and structural constraints on projects in the AGSO sectors.

Box 2: Statistical Analysis After creating a database on all projects, two types of analysis were performed. First, descriptive statisticsfrequencies, percentages, and histogramswere prepared to illustrate the distribution of projects against each indicator. Second, multivariate regression analysis using the ordinary least square method was performed to assess relationships between each indicator and the performance measures. Two sets of multivariate regression models were built. The models were based on a conceptual framework comprising the major participants in a project. Factors relating to participation by beneficiaries and the environment could not be included (para. 9). The first set of models focused on identifying the distinction between the typical factors that affect project performance in AGSO and ENIN sectors. The second set assessed the significance of typical as well as other factors specific to AGSO sectors. The first set of models included explanatory variables relating to the ?? Banks performance at project preparation: presence of project preparatory technical assistance (PPTA) for design; ?? Banks performance during implementation: supervision in person-days; ?? executing agencys (EAs) performance during implementation: time and cost overruns/underruns; and ?? borrowers performance: long-term country economic performance and mean growth rate of gross domestic product. The second set of models included explanatory variables relating to the ?? Banks performance at project preparation: rating on quality of design; ?? Banks performance during implementation: rating on adequacy of supervision; ?? performance of consultants during implementation: rating on adequacy of consulting services; ?? EAs performance during implementation: rating on adequacy of EA capability to supervise implementation; ?? EAs performance during operation: rating on adequacy of utilization of facilities, and rating on adequacy of repair and maintenance; ?? borrowers performance: composite indicator representing a satisfactory policy environment for the project, commitment to the project, and provision of domestic funds; and ?? external factors: composite indicator on the presence of recession, political unrest, and inclement weather during project implementation, and qualitative index on change in prices between appraisal and postevaluation. The equations and the results of the regression analyses are in Appendix 3.

C.

Limitations of the Study

9. The study is a pilot attempt to employ statistical methods in analyzing the performance of projects. The analysis relies on information from PPARs. In the models, only economic internal rates of return (EIRRs) could be easily incorporated as dependent variables.

8 Consequently, although the study included 141 projects88 projects in AGSO sectors and 39 in ENIN sectorsthe models included only 34 cases from AGSO sectors and 31 cases from ENIN sectors. Of the 34 cases from AGSO sectors, only 3 social infrastructure projects were featured. However, the database on the typically monitored indicators included all projects in both AGSO and ENIN sector groups, while that on other indicators specific to AGSO sectors covered more than 75 percent of the projects in these sectors.1 Information constraints made it difficult to analyze the impact of such variables as the participatory approach and the environment, as the number of PPARs referring to them is still limited. Therefore, the Mission exercised caution in interpreting the results of the regression models and drawing conclusions. In the early stages, the Economic Analysis and Research Division (EDAN) provided comments and expertise on the technical aspects of the regression analysis used in the study, which ensured that the analysis was correctly conducted and the inferences drawn followed accordingly. In summary, the findings of the study as a pilot attempt to statistically analyze the performance of Bank-assisted projects produced meaningful (Appendix 3, Tables 6 to 9) and understandable results. D. Finalizing the Report

10. As this study is a pilot attempt, the Mission organized an inception workshop where the methodology and approach for the study were clarified. 11. The final report duly reflects comments received at another workshop, and those from operations departments/offices on the interdepartmental draft. II. MAJOR FINDINGS OF THE STUDY

A. 12.

Performance Measure The regression models used two dependent variables: (i) the REIRR as reported in the PPAR (in cases when there was a range of values for REIRR, the lowest value was used); and the change in EIRRs between appraisal and postevaluation, represented by the percent gap in EIRRs (PGIR).2

(ii)

The data constraints precluded logit-modeling using ordinal dependent variables such as performance rating and achievement of objectives. Limitations of time and resources also disallowed quantile-regression for analyzing distinct characteristics of outstanding and poor performing projects. PGIR = (EIRR at postevaluation EIRR at appraisal) x 100. EIRR at appraisal

9 13. EIRRs were estimated in 65 cases (46 percent of all cases) at both appraisal and postevaluation. Nearly all projects in ENIN sectors and about 50 percent in AGSO sectors had REIRRs.1 14. The REIRR represents the efficiency of the investment and is a major yardstick of project performance/rating at postevaluation. Generally, when the REIRR is above 10 percentwhich is the usual EIRR benchmark at appraisal as wella project is rated as generally successful. A project is rated as partly successful when the REIRR is between 4 and 10 percent, and unsuccessful when REIRR is below 4 percent. 15. The model focused on identifying the direction and magnitude of each explanatory factors potential impact. Typically, the question raised was whether the impact would be large enough to effect a change in the performance rating one or two categories up or down (e.g., from partly successful to successful). 16. The EIRRs at appraisal varied considerably from the REIRRs (Appendix 1, Table 4). Disregarding external factors, when a project is appraised well, is implemented successfully, and operates efficiently, the gap in EIRRs should be zero. 17. A review of the data on the REIRR and the PGIR (Appendix 1, Tables 3 and 4) yielded two findings on project performance in AGSO sectors. (i) Projects in AGSO sectors are performing less satisfactorily than those in ENIN sectors. Projects in AGSO sectors have low REIRRsbelow 10 percent in 20 of 35 cases, with an average of 10.5 percent2 (Appendix 1, Table 3). The average of REIRRs for projects in ENIN sectors, at 18.5 percent, is well above the benchmark for a rating of generally successful.3 The EIRRs at appraisal differ considerably from the REIRRs. The data suggest that appraisal in both sectors was biased. In AGSO sectors, less than half the level of the EIRR estimated at appraisal was achieved at postevaluation.4 In ENIN sectors,5 where the gap in EIRRs was smaller, about two thirds the level of the EIRR estimated at appraisal was achieved at postevaluation.

(ii)

2 3

Projects in Agriculture and Support Services (AGSS), Water Supply and Sanitation (WSS), Education, and Health and Population (HAP) sectors were not featured, as there were few or no cases with REIRRs. Economic analysis was not required by the Bank for projects in these sectors during that approval period. REIRRs ranged from 0 to 18 percent, except three cases, which had REIRRs above 20 percent. In nearly 25 of 34 projects, REIRRs were above 10 percent. In three cases, REIRRs were higher than 35 percent. In AGSO sectors, PGIRs ranged from minus 88 percent to plus 38 percent. Except in four cases, PGIRs were below zero, indicating that REIRR is below EIRR at appraisal. In ENIN sectors, PGIRs ranged from minus 100 percent to plus 100 percent; in one case, it was plus 300 percent; PGIRs were positive in 9 of 31 cases. In the case of projects in the energy sector, such differences between the EIRRs at appraisal and the REIRRs occurred mainly because the assumptions at appraisal relating to increases in tariffs were not fully achieved due to sociopolitical reasons.

10 B. Typically Monitored Indicators of Project Performance

1.

Delays

18. In the study, time variations were significant in both AGSO and ENIN sectors. The average delay was about 2 years (Appendix 1, Table 5). Projects in Water Supply and Sanitation (WSS), Health and Population (HAP), and power sectors had long delays of over 40 months.1 A trend of increasing delays by evaluation year was observed (Appendix 1, Table 6). It may be appropriate to anticipate an average of 2-year delay and reflect it in the sensitivity analysis in the economic evaluation in Reports and Recommendations of the President. 19. With respect to completion on time or ahead of schedule, the proportion of projects was high in ENIN sectors28 percent compared with 15 percent in AGSO sectors. 20. The magnitude of time overrun was generally greater for projects in ENIN sectors. The reason could be that estimates of their implementation periods at appraisal were unrealistic. The average implementation period at appraisal was an ambitious 2.75 years in ENIN sectorsmuch below the same for projects in AGSO sectors, which was 4.73 years, and the average actual implementation period of 4.66 years for projects in ENIN sectors. Understandably, nearly one third of projects in ENIN sectors experienced more than 100 percent time overrun (Appendix 1, Table 7). 21. According to the model analysis, other things being equal, every unit of increase in time overrunmeaning 100 percent delay in completion, would result in reduction by 2.2 percentage points in the REIRR, and a deviation by minus 11 percent from the EIRR at appraisal. The coefficients were zero for AGSO sectors, implying that managing delays alone would not improve project performance measured in terms of REIRRs. 2. Cost Underrun/Overrun

22. Cost underruns were more common than overruns in both sector groups. The underruns were greater in projects in AGSO sectors.2 23. The cost underrun/overrun affected the REIRR significantly only in ENIN sectors. The REIRR model revealed that, other things being equal, the average impact of 50 percent overrun is to decrease the REIRR by 1.6 percentage points. The coefficients were zero for AGSO sectors, implying that even substantial cost underruns would not significantly improve project performance in terms of REIRRs.
1

Overall, 14 projects in AGSO sectors (7 in ENIN sectors) had delays of over 40 months. Another 16 in AGSO sectors (4 in ENIN sectors) had delays of over 30 and up to 40 months. Some reasons cited in the PPARs for such cost underruns were (i) devaluation of the local currency; (ii) cost cutting in the purchase of equipment or materials, especially in the education sector. Only 22 of 88 (25 percent) projects in AGSO sectors had cost overruns, while 18 of 37 (48 percent) projects in ENIN sectors had overruns (Appendix 1, Table 8).

11

3.

Bank Supervision

24. Bank supervision, in person-days, was on average above 100 person-days for a project in AGSO sectors; however, it was above 100 person-days in only 13 of 48 projects in ENIN sectors (Appendix 1, Table 9). The input variable, person-days spent on project review, however, was inadequate to assess whether Bank supervision was effective or not. In the models, the coefficients for Bank supervision were zero for both AGSO and ENIN sector groups.1

Adequacy of Bank supervision was rated only for projects in AGSO sectors (Appendix 1, Table 10). The average of ratings was between generally adequate and partly adequate. However, according to the models, the marginal impact of adequacy of Bank supervision on EIRRs was not significant.

12 C. Indicators of Project Performance Specific to AGSO Sectors

1.

The Banks Performance

25. The Banks performance was found important for project performance. The following variables were used in the models: (i) (ii) presence or absence of PPTA, and qualitative rating on the project design. a. Presence of PPTA

26. Overall, 93 of 141 projects were preceded by PPTA (Appendix 1, Table 11). Of the 93 projects, 45 percent were rated as generally successful, 42 percent as partly successful, and 10 percent as unsuccessful (Appendix 1, Table 14). On the positive side, 85 percent of the generally successful projects in AGSO sectors were preceded by a PPTA.1 This partially supports the Banks process cycle, which normally involves a PPTA for a loan project. However, among the AGSO projects preceded by a PPTA, a considerable number (64 percent) were either partly successful or unsuccessful. It appears that a PPTA is largely necessary for project success, but is by no means sufficient.2 27. First, PPTAs did not guarantee the outcome of projects in AGSO sectors (Appendix 1, Tables 12 and 13).3 Evidently, the average of REIRRs for projects with PPTA fell sharply, even below those without PPTA. This raises a serious concern on the quality of economic analysis at appraisal, presumably based on the PPTA.4 28. The coefficient for the presence of a PPTA, was minus 5.2 in the REIRR model, meaning that the average REIRR for projects that were preceded by PPTA was 5.2 percentage points lower than the average for projects with no PPTA. 29. Second, the PGIR model revealed that, other things being equal, the presence of a PPTA resulted in huge negative gaps in EIRRs, suggesting an overestimation of economic impacts at appraisalprobably assuming conditions that could not be actually achieved during implementation.5 This is especially applicable to partly successful projects; the average of
1 2

For ENIN sectors, the value was 72 percent. The Projects Department suggested that projects without PPTAs may perform better because they are simpler in design and may be repeats of previous projects. The overall performance of projects with PPTA in AGSO sectors during the approval period 1981-1987 exhibited a declining annual trend (Appendix 1, Tables 14 and 15). For projects that were preceded by PPTA, the averages of EIRRs were 25.6 percent at appraisal, 18.7 percent at project completion, and 10.3 percent at postevaluation, respectively; however, similar values for projects that were not preceded by PPTA, were 21.3 percent, 15.1, and 14 percent, respectively. The coefficient was minus 45 in the PGIR model, meaning that the average negative deviation in REIRRs from EIRRs at appraisal for projects with PPTA was 45 percent compared with that for projects without PPTA. As a result, the average PGIR for all projects in AGSO sectors was very low at minus 61 percent.

13 EIRRs at appraisal was even higher for these projects than for those that were rated as generally successful.1 30. As an additional note, the presence of a PPTA was also strongly associated with time overruns, implying that PPTAs were not effective in ensuring timely implementation either. b. Quality of Design

31. The quality of design was rated high, partly adequate, and low (Box 3). In about 53 percent of the cases, the rating was only partly adequate (Appendix 1, Table 16). More specifically, such cases were from Irrigation and Rural Development (IRAD), fisheries, and agricultural and support services (AGSS) sectors. Incidentally, the data showed no statistical evidence of linkage between high quality of design and the presence of PPTA. For example, for the total of seven projects in AGSS sector that were preceded by PPTA, the quality of design was partly adequate for five cases and low for the rest. Of the remaining eight AGSS sector projects not preceded by PPTA, four scored high in quality of design.
Box 3 High Quality of Design Design is detailed, well prepared, and flexible for possible modifications; design used appropriate approach and has appropriate components; and design include several cost savings measures (PEO Nos. 352, 358, 457, 486, 491, 501, 337, 396, 429, 484, 346, 430). Low Quality of Design Limited analysis of policies and institutional framework; inappropriate assessment and provision of working capital; inflexibility and limited focus; and insufficient or no consideration of sociocultural aspects (PEO Nos. 350, 358, 370, 387, 393, 458, 486).

32. The coefficient for quality of design at entry was 2.9 in the REIRR model, meaning that the average impact of a unit improvement in the quality2 was to increase REIRR by 2.9 percentage points. Other things being equal, if the quality of design would improve to high for projects in AGSO sectors,3 the average of REIRRs would go up from 10.5 percent to 13.5 percent; more projects would qualify for the generally successful rating (Appendix 1, Table 16). 2. Performance of the Consultants During Implementation

2 3

The average EIRR at appraisal was 27.7 percent for partly successful projects with PPTA and 25.5 percent for generally successful projects with PPTA; the corresponding averages of REIRR were 5.9 percent and 14.9 percent, respectively (Appendix 1, Table 12). The pattern is the same for the without PPTA case. For example, the rating improves from low (rating = 0) to partly adequate (rating = 1). The project-weighted average of ratings was 0.95. The improvement in quality, to high in all projects, will result in an increase of 1.05 units in the average rating, from 0.95 (current average) to 2 (rating for high quality). The resultant impact on the REIRR is 1.05 times the coefficient (2.9), which is 3.0 percentage points.

14 33. The adequacy of consulting services was rated qualitatively (Box 4). Statistical analysis revealed that when the quality of PPTA was high, the performance of consultants was also adequate during implementation for all AGSO projects.1

The analysis identified a positive correlation significant at a probability of rejection of 1 percent. Further review of individual projects in this regard was not undertaken.

15

Box 4 Adequate Consulting Services Performance of consultants was consistent with the terms of reference; efficient consulting services with respect to site investigation, design submission and construction supervision; on-schedule accomplishment of work; acceptable development scheme and successful transfer of new technology (PEO Nos. 390, 399, 412, 422, 439, 451). Inadequate Consulting Services Insufficient professional expertise on sector/ country development; slow performance of consultants and incapability to adapt to changes; deficiencies in coordinating and scheduling of project activities; no undertaking of relevant surveys; poor working relationship with EA staff; limited understanding of beneficiaries sociocultural traits; and overlooking of important qualifying considerations and technical aspects (PEO Nos. 354, 384, 357, 375, 493, 463, 443).

34. The coefficient for adequacy of consulting services was 2.5 in the REIRR model, i.e., every unit increase in the rating on adequacy of consulting services would have raised the REIRR by 2.5 percentage points. For example, if the rating for adequacy of consulting services, which was only partly successful in several projects in IRAD sector, improved to adequate,1 the average of the REIRRs would have been higher by 4.5 percentage points: from 9.1 percent to 13.6 percent. There would have been more projects with the rating of generally successful (Appendix 1, Table 18). 3. 35. 36. (i) (ii) (iii) Performance of the Executing Agency The EAs performance was found to be even more important for project outcome. The following variables were used in the analysis: qualitative rating on capability to supervise project implementation, qualitative rating on utilization of project facilities, and qualitative rating on maintenance and repair of project facilities. a. Capability to Supervise Project Implementation

37. Nearly 65 percent of the EAs in country group A were rated incapable or partly capable, compared with only 39 percent in group B countries2 (Appendix 1, Table 19). Box 5
1

The average rating was at 1.2. The improvement would result in a REIRR equivalent to 1.8 times the coefficient. Group A countries are Afghanistan, Bangladesh, Bhutan, Cambodia, Peoples Republic of China, Cook Islands, India, Kiribati, Kyrgyz Republic, Lap Peoples Democratic Republic (Lao PDR), Maldives, Republic of Marshall Islands, Federated States of Micronesia, Mongolia, Myanmar, Nepal, Pakistan, Solomon Islands, Sri Lanka, Tonga, Tuvalu, Vanuatu, Viet Nam, and Western Samoa; Group B, Indonesia, Kazakhstan, Republic of Nauru, Papua New Guinea, Philippines, and Thailand; and Group C, Fiji, Hong Kong, Republic of Korea, Malaysia, Singapore, and Taipei, China.

16 gives some characteristics of EAs with adequate and inadequate capability. The coefficient for capability of EA was 4 in the REIRR model, so that the average effect of every unit increase in the rating is to increase the average of REIRRs by 4 percentage points. For example, if EAs for projects in IRAD sector were to perform as well as those for projects in the urban development and housing (URD) sector,1 the average of REIRRs would go up from 9.1 percent to 12.7 percent.
Box 5 Adequate Capability of EA Well-qualified, committed, and wellequipped staff for project management; smooth/effective coordination between EAs; sound working relationship with beneficiaries, consultants, and Bank staff; compliance with project-specific covenants; prompt appointment of project personnel; appropriate organizational structure; strong capacity to run continuing programs and market projects (PEO Nos. 332, 410, 418, 419, 434, 468, 478, 348, 461, 480, 494, 500). Inadequate Capability of EA Reliance on external funding; insufficient interest among EA staff, frequent staff turnover and untimely appointment of key management staff; deficient compliance with project-specific covenants; inadequate expertise in specific project/sector data analysis; inability to overcome weakness of data or to reassess the fundamental assumptions of project design; inappropriate benefit monitoring and evaluation and absence of specific organizational unit to monitor; nonfamiliarity with procurement procedures; weak accounting and reporting systems (PEO Nos. 330, 333, 388, 452, 398, 477, 400).

b.

Utilization of Facilities

38. The average of ratings was 1.1, between moderately utilized and highly utilized (Appendix 1, Table 17).2 The analysis revealed that in cases where facilities were rated highly utilized, the EA was rated capable. It was also found that in such cases, the performance of consulting services were also adequate. Box 6 gives some reasons for nonutilization of project facilities.
Box 6 Highly Utilized Project Facilities Most facilities were fully/well utilized; and major modifications of facilities allowed satisfactory operation (PEO Nos. 352 and 430). Unutilized Project Facilities Facilities unutilized due to budget constraints; untimely use of facilities; improper use of credit facility due to insufficient collateral; unutilized facilities due to poor location and poor physical working condition (PEO Nos. 458 and 486).

1 2

The average of ratings for capability of EA was 1.7 for IRAD sector and 2.6 for URD sector. 0 = facilities not utilized, 1 = moderately utilized, and 2 = highly utilized.

17 39. The coefficient for utilization of facilities was 3 in the REIRR model, so that every unit increase in the rating will increase the average of REIRRs by 3 percentage points. In the PGIR model, the coefficient for utilization of facilities was 11. Other things being equal, for example, if utilization of facilities in projects in IRAD sector1 improved to highly utilized, then the average of REIRRs would improve from 9.1 percent to 11.8 percent. c. Maintenance and Repair

40. Projects in country group A suffered from poor maintenance, while those in group B fared better (Appendix 1, Table 20). Information was available for only 60 percent of the cases.2 Maintenance was rated inadequate/partly adequate in most projects.3 It appeared that maintenance and repair were not distinctly satisfactory even for outstanding projects in AGSO sectors. Maintenance was rated adequate in only 7 percent of the cases. Consequently, in the models, the coefficients for this indicator were not significant. Considering the close linkage of adequate maintenance with sustainable and efficient operations, the Bank should place greater emphasis on ensuring adequate maintenance and repair of projects in AGSO sectors. 4. Performance of the Borrower

a.

Qualitative Rating

41. The borrowers performance is represented by a composite index signifying the presence of supportive policies, adequate domestic funding, and commitment to the project (Box 7). It was found to be the most important indicator of project outcome.4 42. In 86 percent of the cases, the borrowers performance was considered satisfactory. Such cases were equally distributed among country groups A and B (Appendix 1, Table 21). The borrowers performance was unsatisfactory in all projects rated as unsuccessful.
Box 7 Satisfactory BorrowersPerformance Successful privatization of product distribution; provision of adequate budget; encouragement of increases in school enrollment; promotion of aquaculture and inland fisheries development; priority to measures to reduce incidence of disease
1 2

Unsatisfactory Borrowers Performance Little progress in reforms; government policies inconsistent with project goals; lack of support for policy changes by government bureaucracy; inadequate attention to repair deteriorated project facilities and funds for regular maintenance; withdrawal of state

The average rating was 1.1. Maintenance and repair were rated 0 when inadequate, 1 when partly adequate, 2 when generally adequate, and 3 when adequate. The averages of ratings were partly adequate (just above 1.0) in IRAD, forestry, and URD sectors. Averages were below a worrisome 0.5 in education and HAP sectors. A qualitative rating on performance was used: zero = unsatisfactory performance, and 1 = satisfactory performance.

18
and to protect the environment (PEO Nos. 443, 358, 370, 457, 458, 486, 491, 499, 337, 396, 429, 484, 346, 432). patronage from inland freshwater fishery activities; and termination of project activities and reduction of project scope (PEO Nos. 443, 451, 390, 463, 493, 375).

43. The coefficient for borrowers performance was 7 in the REIRR model, so that the average effect of satisfactory performance was to improve REIRR by 7 percentage points. In other words, unsatisfactory borrowers performance as opposed to general expectations at appraisal alone would result in a decline in the REIRR by 7 percentage points; the average of the REIRRs would be 3.5 percent rather than 10.5 percent: most projects would be rated as unsuccessful. b. Country Economic Performance Index

44. The models used an index of country economic performance developed by WB1 (Appendix 1, Table 22). On a scale of 1 to 5, the index represents the average country performance rating for the period 1979 to 1992. This covers the approval periods (1981 to 1992) for cases in this study. 45. The averages of the REIRRs for country groups A and B were 14.7 percent and 15.4 percent, respectively. The average of PGIRs for both groups was minus 27 percent. 46. In the REIRR model, other things being equal, if Nepal, the country with the poorest performance rating (2.73) mimicked Thailand, the country with a high performance rating (4.13), the average of REIRRs for Nepals portfolio would significantly improve by 15 percentage points. This suggests that the Bank needs different policies to help countries like Nepal, which face substantial challenges that impede development. 47. The specific significance of the index in AGSO sectors could not be perceived because of constraints, which concentrated data in only some countries.2 c. Country Policy Environment and Institutional Quality

48. The study supported previous findings that the quality of a countrys macroeconomic policy environment is an important factor in differing performance within its project portfolio.3 The study used the WBs classification of DMCs in a 2x2 matrix by policy

2 3

The index rates countries based on separate ratings for equally weighted components: short-term economic management, long-term economic management, poverty reduction, and responsiveness to WB. The index not only captures countries medium- and long-term structural conditions but also avoids the collinearity problems in various macroeconomic indicators (Source: WB. 1995. Annual Review of Evaluation Reports). All cases in AGSO sectors were crowded among five countries, whose indices ranged between 3 and 3.8. The 1997 World Development Report, The State in a Changing World, finds that sound macroeconomic policy and capable state institutions have large impacts on a countrys long-term growth performance and on the effectiveness of public sector investments. This was confirmed in another WB study (WB. 1997. Annual Review of Development Effectiveness (ARDE).

19 performance and institutional quality (in other words, governance).1 The average performance of the Banks portfolio was calculated for each country and group of countries.2 49. The average project portfolio performance, for the group of countries with both high policy performance and high institutional quality was 0.84 (as against 1.00 for the best performance) compared with 0.72 for the group with low policy performance, but partly adequate institutional quality (because there are no low countries among Bank DMCs) (Table 4).
Table 4: Country Classification by Policy Performance and Institutional a Quality, and Average Project Portfolio Performance
Policy Performance Medium Country APR

Institutional Quality High

High Country Thailand Malaysia Group Average

Low Country APR

APR 1.00 (8) 0.73 (11) 0.84

Partly Adequate Low Indonesia Philippines Group Average 0.7 (25) 0.6 (15) 0.66 Bangladesh 0.7 (15)

Pakistan Sri Lanka Group Average

0.75 (14) 0.68 (11) 0.72

APR = average project portfolio performance. The number of projects in each country is shown in parentheses. The group average is the project-weighted average for all countries in a group.

50. The interrelationship between best policy and effective institutions also suggests that getting institutions right brings significant gains not only in portfolio performance but also in the future stability of the policy environment. 5. External Factors

51. The effect of external factors on project performance was also observed (Box 8). During project implementation, prices decreased in several cases (Appendix 1, Table 23). The study was limited to analyzing the statistical significance of the impact of this important factor on the REIRR or the PGIR, as more than half the PPARs (56 percent) did not report on this factor explicitly. Projects in WSS, IRAD, and AGSS sectors suffered from other external effects such as political unrest, inclement weather, and recession (Appendix 1, Table 24). Due to data constraints, the models did not reveal any statistically significant impact of price decreases on the REIRR or the PGIR.
1

The measure of policy performance is a project-weighted index based on three principal components: inflation, fiscal balance, and openness. Institutional quality, which is also project-weighted, is a composite of measures of quality of government: the extent of red tape involved in any transaction, the regulatory environment, and the degree of autonomy from political pressure (Source: WB. 1997. ARDE). The average project portfolio performance is the project-weighted average of performance rating (in which generally successful = 1, partly successful = 0.5, and unsuccessful = 0) of a countrys project portfolio.

20

Box 8 Significant External Factors Change of government in 1986 created uncertainty; political attention focused on the possible privatization of the units complexes; intermittent civil disturbances; catastrophic floods in 1987-1988 affected loan recovery rate; occurrence of a hazardous fire; economic recession; and major political upheaval and peace and order problems (PEO Nos. 337, 346, 350, 354, 384, 412, 430, 435, 443, 463).

III.

CONCLUSIONS AND IMPLICATIONS

52. The study faced various data constraints and other limitations (para. 9), including a restricted number from social infrastructure projects. It yielded indicative results and reinforced lessons already known but not fully addressed in the Bank.1 53. The study confirmed that monitoring the project implementation schedule and disbursement levelswhich in effect means, managing time and cost overrunswas insufficient to improve the performance of projects in AGSO sectors. Such factors appeared more important in ENIN sectors. 54. For AGSO projects studied, equal attention should have been paid to other factorsboth project specific and externalwhose impact on project performance was significant. This supports the recent effort in the Bank to introduce a broader, more impactoriented approach to project implementation monitoring and management based on the project performance report. 55. Regression models identified such factors as macroeconomic performance and policy and governance to be critical determinants of performance in AGSO sectors. At the country level, the Bank needed to assess more realistically the macroeconomic and policy environment, and choose a set of interventions that addressed existing constraints. At the project level, in countries where the policy framework was distorted and the implementation environment was weak, necessary advisory and capacity building services were frequently needed to precede lending activities. 56. The study confirmed that the borrowers performance was the most critical determinant of project success. Good performance? which implies a policy environment favorable to project operation, government commitment to the project, and provision of domestic funds? could improve project performance significantly. 57. After borrower performance, project success in AGSO sectors depended on the EAs capability to supervise project implementation and properly utilize facilities, and its commitment to maintenance and repair. The Banks continuing focus on encouraging
1

The 1998 Annual Performance Evaluation Program included similar findings.

21 commitment to ownership of project goals by increasing EA involvement in design and implementation is well underscored. 58. Statistical results show that adequate consulting services during implementation yielded significant rewards by sharply reducing the gap in EIRRs between appraisal and reevaluation. 59. The Banks foremost contribution can come at the PPTA stage where, in particular, greater realism could be required. The PPTAs included in this study did not generally ensure satisfactory project design or adequately guard against overestimating the EIRRs. The study suggests a possible need to undertake a critical review of Bank policies and practices relating to PPTAs with a view to enhancing their positive contribution to project performance.

22

APPENDIXES

Number

Title

Page

Cited on (page, para.)

Statistical Tables

18

4,8

Bibliography

33

4,8

Database on Indicators and Regression Analysis

34

4,8