Outcomes from 2009-10 impact assessments

In 2009–10 five impact assessments were finalised and reports published. Two more have been completed and are due to be published in early 2010–11. The focus on undertaking thematic impact assessments, rather than assessing the impact of individual projects, continued, with assessment of ACIAR’s investment in research on forestry and forages in Indonesia and forestry in PNG. In addition, an analysis of studies reporting the returns to investment in the CGIAR, focusing on ACIAR’s mandate regions, was completed and published.

This is the seventh year in which adoption studies were undertaken. The adoption study portfolio now contains 65 sets of projects from which we are able to learn important lessons that are fed back into our project development, design and implementation process. Reports in ACIAR’s adoption studies series also provide an estimate of adoption characteristics for the projects covered. The qualitative assessment suggests that, in the majority of cases, the project outputs are being used by either intermediaries or the final targeted groups. While the measure of adoption provided in the studies is relatively subjective, seven projects reporting a high level of uptake have been subjected to a more in-depth quantitative analysis. With one exception, each was found to have significant benefits.

The experience gained through all these activities has been used to provide two broad types of training workshops. The first type, held in Quy Nhon in Vietnam in 2009–10, focused on withinproject evaluation aimed at providing researchers involved in ACIAR projects with the skills required to ensure that projects are designed and managed in a way that recognises the complex pathways to change. The aim of the second type of training workshop was to provide agricultural economists with the framework and skills necessary to undertake detailed quantitative impact assessments. This training workshop was co-facilitated with IRRI and held at IRRI headquarters in the Philippines. In addition, ACIAR in general, and the Impact Assessment program in particular, provided considerable input into the strategic review of Australia’s rural development assistance conducted by the AusAID’s Office of Develoment Effectiveness. The aim of the review is to guide actions that will strengthen the impact of Australia’s official development assistance in rural areas.

Impact assessments

Benefit-cost meta-analysis of investment in the International Centres of the CGIAR

The Australian Government has supported International Agricultural Research Centers (IARCs), primarily the Consultative Group on International Agricultural Research (CGIAR) since their founding in the early 1970s. Since 1992, this support has largely been channelled through ACIAR. For core funding, the priorities are to foster strong linkages between IARCs, ACIAR and Australian research organisations working together in our partner countries, and to contribute to strengthening of IARC governance to ensure that programs focus on rural poverty and inclusive growth. The priorities for project-specific funding are to strengthen the link between Australian research institutions, NARS and the IARCs, and to help focus IARC institutions on ACIAR’s bilateral priorities.

The key objective of this study was to assess and compile evidence of how effective CGIAR investment in agricultural research and development (R&D) has been in terms of achieving economic impacts within ACIAR’s mandate regions. In doing so, this analysis required the selection of completed impact assessments providing large-scale and verifiable results. As such the aggregated benefits from this selection process represent a lower bound of benefits that have been confidently realised as it comprises only those studies for which reliable and credible economic impacts have been published and where delineation of the benefits to ACIAR’s mandate regions could be made. In addition, it was not possible to disaggregate expenditure by the individual CGIAR centres into investments inside or outside the ACIAR mandate regions. Hence, while this process will not capture all the economic benefits arising from investment in agricultural research in the mandate areas, and is likely to overstate the costs, it provides a degree of confidence that the benefit:cost ratio reported is a reliable base measure.

Implementing this procedure indicates that, under the most restricted set of assumptions about credible benefits, every $1 million invested by the CGIAR system in the ACIAR mandate regions produces a return to the developing countries in these regions of at least $2.7 million. Under a more relaxed set of assumptions about credible, plausible and potential benefits the return is up to $3.9 million.

The biology, socioeconomics and management of the barramundi fishery in Papua New Guinea’s Western Province

From July 1999 to December 2003, ACIAR provided funding for a project on the biology, socioeconomics and management of the Barramundi fishery in the Fly River and adjacent coast of PNG. The primary aim of this project was to draft a barramundi fishery management plan that was acceptable to all stakeholders. The plan was passed into PNG law on 15 April 2003 and was gazetted by the PNG Government the following day. This in itself represented a change in practice by PNG policymakers and was a significant achievement of the project.

While the project made a significant contribution to the knowledge required to successfully manage the Western Province barramundi fishery, the barramundi fishery management plan and, therefore, the project do not appear to have delivered any significant benefits to the community. Several other factors seem to have prevented the barramundi fishery management plan from delivering greater benefits to the community: inadequate enforcement; failure to deal with the problem of over-fishing with lures; and a total allowable catch that appears to be too high and not necessarily enforceable. Nevertheless, the project may yet deliver some significant benefits to the community in the future if its scientific outputs are used to underpin revisions to the BFMP.

Extending low-cost fish farming in Thailand: an ACIAR–World Vision collaborative program

In an attempt to increase adoption of the results of earlier technical research projects, ACIAR invested in a collaborative program of extension-based projects with World Vision (WV). The program was composed of six projects located in Thailand, Lao PDR and Vietnam. The ACIAR-WV program consisted of a combination of participatory research and extension training. The aim was to use WV field staff engaged in community development projects to direct the research and deliver the extension advice. Impact assessment of all six components of the ACIAR-WV program was not feasible because there was insufficient information. An alternative approach was to select a project component for an impact assessment as an indication of the value of this type of investment. The Thai fish-farming project was selected for this purpose. The present value of the net benefits of the project was $6.9 million for a 5% discount rate. The project had a benefit:cost ratio of 5.1:1. Attribution of the net benefits based on project expenditures indicates a net benefit of $0.9 million for the ACIAR investment.

ACIAR investment in research on forages in Indonesia

Six related ACIAR Indonesian forage research projects focused on increasing the adoption of productivity-improving technologies and thus the incomes and livelihoods of crop–livestock smallholders in eastern Indonesia. An impact assessment was undertaken to estimate the total returns to this research and to attribute part of these returns to ACIAR.

Expressed in net-present-value (NPV) terms, the adoption of the forage R&D technologies could return an estimated $1,300 million if there is strong policy support and about $1,000 million in the absence of such support. Virtually all (93%) of the gain would flow to producers since the impact on the consumer price is quite small, reflecting the highly elastic demand for cattle faced by producers. The investment, noting that major adoption has yet to occur, appears to offer a high rate of return. The internal rate of return (IRR) is estimated at around 20% and the benefit:cost ratio at greater than 20:1, even allowing for a significant investment in future extension activities of around Rp5,000 million per annum after 2011 for 30 years and beyond (as either new extension funding or reprioritising existing extension service expenditure). This extension investment, in present value terms, is roughly equal to the total R&D investment to date. If the extension investment is increased to Rp15,000 million per annum to achieve a faster rate of adoption, the investment return could be higher, with an estimated IRR of around 22%.

The ACIAR financial contribution to the R&D investment has been calculated at 60%, and the balance through in-kind or other funding of the research and extension agencies—predominantly CSIRO, the University of Queensland, Balai Pengkajian Teknologi Pertanian and the universities in Indonesia. For the delivery of the estimated benefits, future extension work will be critical. Both investments—R&D and extension—are required to deliver the estimated benefits. With the base level of extension (and resulting 50%/50% R&D/extension contribution), ACIAR’s share of the benefits would be 30%—around $300 million. As the extra benefits arising from additional extension are attributable to only that additional extension, the value of the benefits attributable to ACIAR under this scenario remains at $300 million.

Reform of domestic grain markets in China: a reassessment of the contribution of ACIAR-funded economic policy research

An ex ante assessment of the impact of two ACIAR funded economics research projects dealing with domestic grain market reform in China was undertaken in 2004 and published as an IAS report. ACIAR investment and in-kind contributions from partners for both projects totalled almost $2.6m (in 2002 dollar terms). At the time of the original impact assessment, policy reviews and empirical measures of assistance to the grains industries suggested that the late 1990s were a period of policy retrenchment rather than reform and hence the welfare gains from this program of research were prospective in nature rather than realised.

Subsequently, estimates of nominal rates of assistance to agriculture in China were substantially revised: the actual experience in the markets was one of a continuing reform process, not policy retrenchment. In light of this, the focus of this impact assessment was to recalculate the return to investment in economics research into grain market reform in China in general and to the ACIAR projects in particular. The chief attraction is that the new analysis is ex post rather than ex ante and is consistent with recent views about the nature and extent of reform in the marketing of grains in China.

These revised estimates of are somewhat higher than the original estimates, as might be expected given the faster rate of reform. The benefit:cost ratio for ACIAR’s investment is in the order of 20 – 30:1. However, the original discussion of other measures of success of the projects still stands. The projects were highly likely to have been successful because of authority of the commitment of the Chinese collaborators, the capacity building within Chinese institutions and the strong publications record and ongoing funding of the projects.