Tag Archives: matching

Impact Evaluation in Conservation Science

This is the first blog in a weekly series exploring impact evaluation and building the evidence base in conservation science and policy.

Impact. It’s a term that we hear everyday in the news without batting an eye. Here are some headlines from just today:







But what does the word impact actually mean? How can we really measure the impact of one thing (let’s say, a low fat diet) on another (weight loss)? Aren’t there many other factors at play that could actually be causing the change? What if, while a subject begins a low fat diet, he also starts exercising more and counting calories? Those factors are likely to be important to the measured outcome (weight loss) as well and have to be “teased out” somehow. How can this be done? Scientifically, we use a technical process called impact evaluation. This process attempts to estimate the actual effect of a treatment or intervention (e.g. a drug) on an important outcome (e.g. weight). This approach is used regularly in the medical research community when testing the effects of drugs on patients. Researchers must carefully design the trial so that the populations taking the drug itself and those taking the placebo represent a random cross-section of people. This helps to ensure that the two groups do not differ significantly in other ways that could affect the outcome. For example, if the treated (drug-taking) group in the trial contains a disproportionate number of individuals that have hypothyroidism (which is associated with low metabolism), this will bias the results. The hypothyroidism-group may be inherently less (or perhaps more!) likely to respond to the effects of the drug. Although the drug could indeed have an effect, it would not be effectively measured by trying it on this non-random sub-sample of the population. Through random assignment of participants to the treatment and control groups, researchers ensure that the groups are not inherently different from each other. This way, the groups are fairly comparable to each other and researchers can compare apples to apples.

How a randomized control trial works http://library.downstate.edu/EBM2/2200.htm

How a randomized control trial works. From http://library.downstate.edu/EBM2/2200.htm

The randomized control trial is considered the gold-standard in scientific research. Because of the randomization, the evidence these studies produce provide solid estimates of the impact of treatments (e.g. drugs) on relevant outcomes (e.g. weight loss). However, in conservation and policy evaluation, it is nearly impossible to design a randomized experiment. For instance, would it be possible to test the effects of a Payments for Ecosystem Services plan using a randomized control trial with control and treatment (e.g. paid) groups? This would be a highly inequitable public policy, as the benefits of the payment would only benefit certain people. What about the evaluation of a protected area using randomization? Is it possible to randomly protect some areas and not others and then measure the outcomes? Of course, this approach would politically and practically impossible.

To get around these constraints, conservation scientists have borrowed from econometrics and have implemented approaches known as quasi-experimental methods for impact evaluation. Because we cannot directly experiment with conservation policies, we instead must employ a roundabout approach – hence the “quasi” term. Using sophisticated statistical techniques, conservation scientists attempt to simulate a randomized control trial. These approaches have been applied to evaluate protected areas and community-based conservation projects alike. The main idea is that these approaches attempt to estimate the counterfactual – what would the outcome have been in the absence of the policy?  We cannot directly observe this, so we can use statistical approaches to estimate it. What are some examples of these approaches? I’ll review three here: matching, instrumental variables, and structured equation modeling.

For example, with an approach called matching, treated groups are compared with select control groups that are as similar as possible to the treatment groups. In other words, they are matched up with each other. This matching process is designed to eliminate or reduce selection bias to the extent possible. Selection bias is essentially a converse of randomization: certain localities are more likely to be selected for a policy treatment (such as a protected area). By using matched controls with similar characteristics, the selection bias can be reduced. Imagine a simple matching situation: you want to estimate the impacts of a protected area on forest cover. To select matches, you’ll use covariates – variables that correlate with the treatment assignment of protection and the outcome variable of forest cover. One example of a covariate is distance to roads. (There are many other possible covariates, but we can just explore one in this simple example.) To select the appropriate matched control, simply choose a locality that is unprotected with the closest possible value for the distance to roads.

matching example

In reality, the analysis would include many covariates that would need to match as closely as possible between treated and control groups. Adding more covariates is not only necessary to reduce selection bias, but also may make it more difficult to identify close matches.

A second approach to try is called structured equation modeling (SEM). This method is somewhat similar to matching, as it employs covariates to reduce selection. However, SEM also allows the inclusion of extra components other than the treatment, called mediating factors, that may also contribute to the outcome. Using this method, it is possible to identify mechanisms – exactly how and through what pathways the treatment affected the outcome. In addition, the interactions between mediating variables can be taken into account – this is an advantage of SEM over matching, which leaves out those interactions.


Another approach to consider employs instrumental variables. In this case, you use an instrument – a variable that affects the probability of treatment but does not affect the outcomes except through the treatment. In other words, the instrument is not correlated with unobserved confounding variables. The advantage here is that it reduces selection bias and only focuses on locations where the treatment variable is not “contaminated” with unobserved confounders. However, it’s difficult to identify instruments in practice that actually work. One example of an instrumental variable that has been used in protected area evaluation is the distance to rivers. The idea here is that protected areas tend to be located near rivers; however, the distance to rivers is unlikely to correlate with an outcome variable such as forest loss or change.

instrumental variables

These are just three examples of evaluation approaches that can be used in conservation science. The appropriate methodological approach for your research will depend on important factors, especially the availability of data and scale of analysis.

For more information about impact evaluation in conservation science, check out these useful references:

Baylis, K., Honey-Rosés, J., Börner, J., Corbera, E., Ezzine-de-Blas, D., Ferraro, P. J., Lapeyre, R., Persson, U. M., Pfaff, A. and Wunder, S. (2015), Mainstreaming Impact Evaluation in Nature Conservation. Conservation Letters. doi: 10.1111/conl.12180

Ferraro, P. J. (2009), Counterfactual thinking and impact evaluation in environmental policy. New Directions for Evaluation, 2009: 75–84. doi: 10.1002/ev.297

How do you know that protected areas are actually protecting anything? Use Matching!

This is the sixth in a series of weekly blog posts covering conservation topics with a focus on protected areas and the laws and institutions that support them (or don’t).

The forests surrounding many protected areas are being rapidly cleared or degraded. Shown is recent deforestation for oil palm plantations along the edge of Bukit Palong National Park in Peninsular Malaysia. Photo by William Laurance, http://theconversation.com/are-nature-reserves-working-take-a-look-outside-9432

The forests surrounding many protected areas are being rapidly cleared or degraded. Shown is recent deforestation for oil palm plantations along the edge of Bukit Palong National Park in Peninsular Malaysia. Photo by William Laurance, http://theconversation.com/are-nature-reserves-working-take-a-look-outside-9432

A new methodological approach is gaining traction in the conservation science community. This method is borrowed from the economics literature and has historically been used to analyze the impact of policies on future wages, socioeconomic status, or other economic variables. Today, conservation researchers are beginning to employ this rigorous approach – known as matching – for a variety of purposes. One prevalent application of matching is the evaluation of protected areas, both terrestrial and marine. Why is the matching approach so popular to evaluate protected areas? First of all, the need to rigorously evaluate protected area performance is becoming more widely acknowledged in the literature. The Convention on Biological Diversity includes targets about national designation of protected areas which should be effectively managed. Without monitoring and evaluation of these lands and waters, there would be no way to determine whether management were effective and whether the targets were met. 

Let’s suppose that you are interested in whether the protected areas in your country are preventing the loss of forests. Ok great. Can you simply compare the deforestation rates inside and outside the protected area? Unfortunately, that won’t be completely accurate. The lands inside and outside of that particular protected area might have different types of vegetation, terrain, or soil productivity. Because of these inherent differences, you cannot simply compare the inside to the outside. Ok, what about comparing the protected area before it was established to after? That is a bit better, because it is focused on the same piece of land. However, it doesn’t accurately capture the impact of protection because there may be other factors at play in this location. Perhaps other policies were enacted around the same time that affected deforestation.

inside outside

Ok, so how do we actually isolate the IMPACT of the protected area (the legal establishment itself) on deforestation? One way to do this is to use matching. To conduct matching, the researcher selects a piece of land that is protected and a similar (matching) piece of land that is NOT protected. The protected area is considered the “treatment” and the unprotected area is the “control.” Comparing the treatment and control to each other is a fair and simple way to quantify the benefits of protection. Hence, matching isolates the impact of the protection policy ITSELF and rules out extraneous factors that could affect the results (like other policies, the impacts of different landscape types or soil types, etc).

inside outside

How do you know if two pieces of land (or water) are a good match? Use covariates! Covariates are variables that correlate with the treatment and the outcome. It is necessary to use covariates because the location of protection or deforestation (or whatever outcome variable you are looking at) on the landscape is not random. Protected lands tend to be placed in isolated, mountainous areas with low soil productivity – high and far from development or urban areas. Also, deforestation occurs happen more often in places that are easier to access – closer to roads or other access infrastructure and also closer to the forest edge. By accounting for these factors using covariates (distance to roads, distance to forest edge, etc), the researcher can select well-matched control regions that are actually comparable to the protected area in question. This reduces bias in the analysis and allows you to compare apples to apples.  Further, it is worth noting that when using matching, the magnitude of the impact is likely to be much smaller than the estimate of impact using more traditional approaches simply because the covariates are accounted for.

What are some limitations of the matching approach? Simply put, sometimes you cannot find a perfect match. There may not always exist enough similar parcels on the landscape that are similar in access and topography to your protected area. Also, there is the issue of choosing covariates. How do you know that you have chosen the correct covariates or a sufficient number of them? As more researchers use matching, the scientific community will gain a more refined understanding of which covariates to use and when.

What research frontiers exist for applying matching in conservation? There are many! Matching could be used to evaluate conservation interventions other than protected areas – matching has been applied to quantify impacts of payments for ecosystem services, for example, The method could also be extended to indigenous reserves, community based natural resource management areas, or other area-based interventions. The application of matching to evaluate protected area downgrading, downsizing, and degazettement (PADDD) is an area of research that merits exploration. Using matching can help answer questions like – what is the impact of changing a protected areas’ status on carbon storage for climate mitigation? What is the impact of reducing a protected area’s size on biodiversity? If a protected area’s protection is removed, how is land cover affected? There are endless possible areas of research and exploration that could employ matching.

What are some tools and resources to help with matching? There is a wealth of literature available on matching and impact evaluation – see here, here, and here.  There are also some R packages – MatchIt and Matching are two examples. If you are aware of other resources, post a comment below!