Tag Archives: interventions

Impact Evaluation in Conservation Science

This is the first blog in a weekly series exploring impact evaluation and building the evidence base in conservation science and policy.

Impact. It’s a term that we hear everyday in the news without batting an eye. Here are some headlines from just today:

http://www.bloombergview.com/articles/2015-09-11/human-impact-on-global-environment-may-be-peaking

http://www.bloombergview.com/articles/2015-09-11/human-impact-on-global-environment-may-be-peaking

http://www.usatoday.com/story/news/2015/09/11/house-votes-reject-iran-nuclear-deal-but-action-has-little-impact/72061716/

http://www.usatoday.com/story/news/2015/09/11/house-votes-reject-iran-nuclear-deal-but-action-has-little-impact/72061716/

http://dailycaller.com/2015/09/11/new-data-shows-legalization-had-no-impact-on-teen-marijuana-use/

http://dailycaller.com/2015/09/11/new-data-shows-legalization-had-no-impact-on-teen-marijuana-use/

But what does the word impact actually mean? How can we really measure the impact of one thing (let’s say, a low fat diet) on another (weight loss)? Aren’t there many other factors at play that could actually be causing the change? What if, while a subject begins a low fat diet, he also starts exercising more and counting calories? Those factors are likely to be important to the measured outcome (weight loss) as well and have to be “teased out” somehow. How can this be done? Scientifically, we use a technical process called impact evaluation. This process attempts to estimate the actual effect of a treatment or intervention (e.g. a drug) on an important outcome (e.g. weight). This approach is used regularly in the medical research community when testing the effects of drugs on patients. Researchers must carefully design the trial so that the populations taking the drug itself and those taking the placebo represent a random cross-section of people. This helps to ensure that the two groups do not differ significantly in other ways that could affect the outcome. For example, if the treated (drug-taking) group in the trial contains a disproportionate number of individuals that have hypothyroidism (which is associated with low metabolism), this will bias the results. The hypothyroidism-group may be inherently less (or perhaps more!) likely to respond to the effects of the drug. Although the drug could indeed have an effect, it would not be effectively measured by trying it on this non-random sub-sample of the population. Through random assignment of participants to the treatment and control groups, researchers ensure that the groups are not inherently different from each other. This way, the groups are fairly comparable to each other and researchers can compare apples to apples.

How a randomized control trial works http://library.downstate.edu/EBM2/2200.htm

How a randomized control trial works. From http://library.downstate.edu/EBM2/2200.htm

The randomized control trial is considered the gold-standard in scientific research. Because of the randomization, the evidence these studies produce provide solid estimates of the impact of treatments (e.g. drugs) on relevant outcomes (e.g. weight loss). However, in conservation and policy evaluation, it is nearly impossible to design a randomized experiment. For instance, would it be possible to test the effects of a Payments for Ecosystem Services plan using a randomized control trial with control and treatment (e.g. paid) groups? This would be a highly inequitable public policy, as the benefits of the payment would only benefit certain people. What about the evaluation of a protected area using randomization? Is it possible to randomly protect some areas and not others and then measure the outcomes? Of course, this approach would politically and practically impossible.

To get around these constraints, conservation scientists have borrowed from econometrics and have implemented approaches known as quasi-experimental methods for impact evaluation. Because we cannot directly experiment with conservation policies, we instead must employ a roundabout approach – hence the “quasi” term. Using sophisticated statistical techniques, conservation scientists attempt to simulate a randomized control trial. These approaches have been applied to evaluate protected areas and community-based conservation projects alike. The main idea is that these approaches attempt to estimate the counterfactual – what would the outcome have been in the absence of the policy?  We cannot directly observe this, so we can use statistical approaches to estimate it. What are some examples of these approaches? I’ll review three here: matching, instrumental variables, and structured equation modeling.

For example, with an approach called matching, treated groups are compared with select control groups that are as similar as possible to the treatment groups. In other words, they are matched up with each other. This matching process is designed to eliminate or reduce selection bias to the extent possible. Selection bias is essentially a converse of randomization: certain localities are more likely to be selected for a policy treatment (such as a protected area). By using matched controls with similar characteristics, the selection bias can be reduced. Imagine a simple matching situation: you want to estimate the impacts of a protected area on forest cover. To select matches, you’ll use covariates – variables that correlate with the treatment assignment of protection and the outcome variable of forest cover. One example of a covariate is distance to roads. (There are many other possible covariates, but we can just explore one in this simple example.) To select the appropriate matched control, simply choose a locality that is unprotected with the closest possible value for the distance to roads.

matching example

In reality, the analysis would include many covariates that would need to match as closely as possible between treated and control groups. Adding more covariates is not only necessary to reduce selection bias, but also may make it more difficult to identify close matches.

A second approach to try is called structured equation modeling (SEM). This method is somewhat similar to matching, as it employs covariates to reduce selection. However, SEM also allows the inclusion of extra components other than the treatment, called mediating factors, that may also contribute to the outcome. Using this method, it is possible to identify mechanisms – exactly how and through what pathways the treatment affected the outcome. In addition, the interactions between mediating variables can be taken into account – this is an advantage of SEM over matching, which leaves out those interactions.

SEM

Another approach to consider employs instrumental variables. In this case, you use an instrument – a variable that affects the probability of treatment but does not affect the outcomes except through the treatment. In other words, the instrument is not correlated with unobserved confounding variables. The advantage here is that it reduces selection bias and only focuses on locations where the treatment variable is not “contaminated” with unobserved confounders. However, it’s difficult to identify instruments in practice that actually work. One example of an instrumental variable that has been used in protected area evaluation is the distance to rivers. The idea here is that protected areas tend to be located near rivers; however, the distance to rivers is unlikely to correlate with an outcome variable such as forest loss or change.

instrumental variables

These are just three examples of evaluation approaches that can be used in conservation science. The appropriate methodological approach for your research will depend on important factors, especially the availability of data and scale of analysis.

For more information about impact evaluation in conservation science, check out these useful references:

Baylis, K., Honey-Rosés, J., Börner, J., Corbera, E., Ezzine-de-Blas, D., Ferraro, P. J., Lapeyre, R., Persson, U. M., Pfaff, A. and Wunder, S. (2015), Mainstreaming Impact Evaluation in Nature Conservation. Conservation Letters. doi: 10.1111/conl.12180

Ferraro, P. J. (2009), Counterfactual thinking and impact evaluation in environmental policy. New Directions for Evaluation, 2009: 75–84. doi: 10.1002/ev.297

Nature Needs Half and Looking Beyond Protected Areas

This is the ninth in a series of weekly blog posts covering conservation topics with a focus on protected areas and the laws and institutions that support them (or don’t).

The Yukon to Yellowstone Initiative works to protect lands within a large, transboundary corridor from the Yukon territory to Yellowstone National Park. http://www.canadiangeographic.ca/blog/posting.asp?ID=1353

The Yukon to Yellowstone Initiative works to protect lands within a large, transboundary corridor from the Yukon territory to Yellowstone National Park. http://www.canadiangeographic.ca/blog/posting.asp?ID=1353

What will it take to achieve long-term, sustainable conservation? Think big. One visionary initiative answers this question with a catchy phrase: Nature Needs Half. What does this mean exactly? The vision set forth is to protect 50% of the surface of the planet in order for nature (and subsequently, people) to thrive. Given the realities of accelerating population growth, development, consumption, and a lack of political focus on the environment, this may seem like an impractical goal. However, a vision to protect half of the planet is admittedly powerful. This narrative inspires a “think big” approach which could serve to motivate conservationists and all people as we plan for the future. Protecting half of the planet, especially large wilderness areas like boreal forests and the Amazon, would help store carbon, regulate the climatic and hydrological systems, and preserve species, among other benefits.

Before we accept the Nature Needs Half mantra, let’s consider some of the technicalities involved. First of all, how much of the planet is currently protected? According to the most widely used protected area ledger (the World Database of Protected Areas), globally about 15% of the land and 3% of the oceans are protected. There is a long way to go until we cover half of the planet with protected areas. However, especially for terrestrial jurisdictions, we are not far from reaching Aichi Target 11 of the Convention on Biological Diversity. This target directs signatory nations to protect 17% of their terrestrial and 10% of their marine areas by 2020. These targets are obviously more short term and achievable than a target of 50% coverage. This begs the question: what is the right target to set? 17%? 25%? What about 100%? Of course, international targets are likely to be driven by political, rather than ecological, considerations. There is no “right” answer here.

Even if the Aichi targets are met – how will it ever be possible for half of the planet to be officially protected? There is one approach that may make this goal more achievable. Let’s expand upon what we mean by “protected.” To date, to be officially classified as a protected area and entered into the World Database of Protected Areas, the location must fit the following definition:

“A clearly defined geographical space, recognised, dedicated and managed, through legal or other effective means, to achieve the long term conservation of nature with associated ecosystem services and cultural values.” (IUCN Definition 2008

However, these officially designated lands and waters are not the only places on the planet that are viable for conservation. In fact, many other management and ownership schemes exist which focus on biodiversity conservation – sometimes solely and sometimes in conjunction with other goals – that are not accounted for in official ledgers. Here are just a few examples of these interventions:

1. Conservation easements are privately protected lands, usually held by a land trust. Similarly to nationally protected areas, conservation easements restrict certain types of land use such as extractive activities or development and preserve the area for its scenic, ecological, and/or cultural values.

National Conservation Easements in the US http://conservationeasement.us/

National Conservation Easements in the US. NCED stands for National Conservation Easement Database. Source http://conservationeasement.us/

2. Community Based Natural Resource Management (CBNRM) relies on a bottom-up approach. Within CBNRM schemes, local communities organize to make decisions about resource management and also share benefits.

Mobilizing poor fishers, pioneering innovative methods of transferring lease rights to water bodies to fisher groups, and developing communal resource management systems proved to be invaluable – and replicable. Photo from IFAD http://www.ifad.org/pub/other/cbnrm.pdf

Mobilizing poor fishers, pioneering innovative methods of transferring lease rights to water bodies to fisher groups, and developing communal resource management systems proved to be invaluable – and replicable. Photo and caption from IFAD http://www.ifad.org/pub/other/cbnrm.pdf

3. Military training areas may include large swaths of undeveloped, fenced-in land that provide habitat for biodiversity. Military training areas cover as low as 1% but likely up to 6% of the planet and have the potential to deliver great conservation benefits, especially given the large budget of the military itself (Zentelis and Lindenmayer 2014).

Marine Corps Base Camp Pendelton focuses on conservation programs  http://www.pendleton.marines.mil/PendletonNews/NewsArticleDisplay/tabid/5440/Article/536727/pendleton-home-of-the-avid-hunter.aspx

Marine Corps Base Camp Pendelton focuses on conservation http://www.pendleton.marines.mil/PendletonNews/NewsArticleDisplay/tabid/5440/Article/536727/pendleton-home-of-the-avid-hunter.aspx

Beyond these three, there are many other types of conservation schemes that go “outside the box” of traditional protected areas. After first acknowledging the existence of these diverse interventions, an obvious question arises: are these other conservation schemes as effective as the “gold-standard” protected areas? Although we don’t have a comprehensive picture of the performance of all of these interventions yet, we do have some evidence about the success of “non-traditional” protection strategies.

1. In Costa Rica, Payments for Ecosystem Services schemes increased forest cover. Evidence has shown that Payments for Ecosystem Services (PES) programs – programs that pay landowners to refrain from deforesting their land – worked to increase forest cover in Costa Rica (Arriagada et al. 2012). Costa Rica is indeed famous for its conservation ethic, so it is not possible to generalize these results globally for all PES programs. However, this offers some hope about the potential for nontraditional protected approaches to be effective.

2. Indigenous reserves were just as (if not more) effective than protected areas in Brazil. In one study, deforestation and fire occurred about equally frequently in protected areas and indigenous reserves (Nepstad et al. 2006). However, the protected reserves tended to be located in more remote areas, which suggests that deforestation or fire would naturally occur less often in these areas. In contrast, indigenous areas were created “in response to frontier expansion, and many prevented deforestation completely despite high rates of deforestation along their boundaries” (Nepstad et al. 2006). This suggests that Brazilian indigenous reserves provide extremely strong levels of protection, likely due to stringent enforcement.

3. Sacred sites can be important places for conservation. For example, traditional communities that live in the Zambezi valley of Zimbabwe consider the local dry forests to be sacred. Researchers found that the sacred forests experienced far less conversion than other localities. Notably, the research also revealed that forest loss was higher in locations where traditional leaders felt disempowered as compared to areas where leaders retained power (Byers et al. 2001). This is another piece of evidence that the engagement and empowerment of local communities is critical for effective conservation.

Although we have some evidence, much more research should be done to document and evaluate the performance of these diverse conservation interventions to understand the true reach of conservation action. It would be interesting to compare the ecological performance over time of a locality that has undergone dynamic changes to its governance in the form of different conservation interventions. It would also be interesting to compare the socioeconomic enabling conditions and ecological performances of these various interventions – in other words, what works, where, and why? Evidence of the viability of these interventions could help justify their accounting in official protected area ledgers. Eventually, this could even help us achieve the vision of protecting 50% of the planet and giving Nature the half that she needs.