Disentangling monitoring programs: design, analysis, and application considerations.

Monitoring programs are an essential tool for assessing and informing conservation efforts but the methods used to gather monitoring data directly influence results. This presents a challenge to conservation professionals when deciding on existing data to inform a given question. We illustrate the challenges of using monitoring data by comparing population trends from two large-scale avian monitoring programs in the western United States: the Breeding Bird Survey and Integrated Monitoring in Bird Conservation Regions programs. We used publicly available data to compare trend trajectory between 2008 and 2015 for 148 species across Colorado, Montana, and Wyoming. Trends were inconsistent for 62% of the comparisons, with species having opposite trends in 21 cases. The inconsistencies found within our species comparisons reflect the inherent differences between program sampling design and analytical approach. Periodically revisiting how and why we monitor natural resources is necessary to advance conservation and management as the lessons learned from long-standing programs guide the development of more recent efforts. Our results emphasize that prior to management actions and policy decisions, managers must be aware of both the sampling design and appropriate ecological inference of any monitoring program.

Introduction

Monitoring programs are often used by managers to inform conservation. In addition to providing population assessments, the challenges incurred in early monitoring efforts led to advancements in design and methodology. For example, contemporary occupancy models were developed to account for the imperfect detection of elusive amphibians during surveys (MacKenzie et al. 2002). These methods have subsequently been applied to many other taxa. Monitoring can now support management hypotheses across spatial scales by tracking species distributions and changes in occupancy or abundance through time (hereafter referred to as trends). However, all monitoring programs are not equal with respect to their contributions to either resolve scientific uncertainty or inform conservation decisions. Before deriving biological inferences, it is important to consider how the unique characteristics of any monitoring program influence results.

Two multi‐species monitoring programs provide information on the population status of breeding landbirds across the western United States. The North American Breeding Bird Survey (BBS, 1966–present) is a United States‐wide federally supported effort that relies on an extensive volunteer network (Sauer et al. 2013). A recently initiated monitoring program, the Integrated Monitoring in Bird Conservation Regions program (IMBCR, 2008–present) employs trained observers to monitor breeding landbirds across 13 states in the western United States (Pavlacky et al. 2017). Breeding Bird Survey and IMBCR can provide information on bird populations at similar spatial scales, (Sauer et al. 2013, Pavlacky et al. 2017), and one might expect changes in bird populations to be measured comparably between programs. However, each program employs distinct sampling methods and analytical approaches, which could predispose inconsistent results.

Distinguishing between sampling frame and analytical methods

How sampling locations are selected across a survey area affects trend estimates. For instance, a random placement of surveys with respect to ecological boundaries, can by chance produce aggregated surveys and an unbalanced sampling of a population across habitats. A spatially balanced design can limit bias by ensuring, with a large sample size, the full range of conditions experienced by a species is sampled. Spatial balance can be achieved when survey locations are dispersed evenly over the extent of a static resource or jurisdiction (Stevens and Olsen 2004). BBS employs a grid‐based sampling frame, with each grid cell covering a one degree latitude by one degree longitude area (Bled et al. 2013). Within each grid cell, the starting location and traveling direction of surveys are selected at random using road infrastructure and consist of 50 sampling points spaced 800 m apart (Sauer et al. 2013). IMBCR uses a Generalized Random Tessellation Stratified (GRTS) sampling frame supporting the selection of sampling locations in a spatially balanced manner across multiple management boundaries (Pavlacky et al. 2017). A sampling location is a 1‐km2 plot containing 16 equally spaced survey points (Pavlacky et al. 2017). The GRTS sampling frame allows the number of sampling locations to vary across years within a management boundary of interest, such as a state, while maintaining spatial balance across the boundary (Stevens and Olsen 2004).

Each program also approaches the analysis of population counts differently. A central difference is how these programs address the imperfect detection of individuals or species during surveys. Imperfect detection occurs when individuals or species are present during a survey but not detected, creating bias that can obscure our ability to measure changes in a population or convey changes where none occur (Thompson 2002). BBS uses count data to estimate the annual population status of a species calculated as an index, a description of a population based on the raw number of observed individuals of a species (Sauer et al. 2013). BBS population indices are modeled to account for survey and year‐specific variation in abundances, though without specifically modeling the detection rates of species across surveys (Sauer et al. 2013). Population trends are estimated by BBS as the ratio of abundance indices between the first and last year of a time interval of interest (Sauer et al. 2013). IMBCR provides population densities (birds per 1 km2) across a range of geographic extents (Pavlacky et al. 2017). Available IMBCR abundance products are estimated using conventional distance sampling (Thomas et al. 2010), accounting for the imperfect detection of species on an annual basis (details in Pavlacky et al. 2017). The annual abundance products currently available from IMBCR can be used to estimate population trend (see Trend estimation).

In this paper, we illustrate how the inherent differences between these two programs can generate differing population trends and highlight the complexities of wildlife monitoring data used to support decision‐making for natural resource management. To mirror the decision process of a wildlife manager, we use the current publicly available products of BBS and IMBCR. We compare population trends obtainable from BBS to trends we estimate using annual abundance estimates provided by IMBCR across three U.S. states: Colorado, Wyoming, and Montana. We also examine the land cover composition of each program within each state to determine if any categories of land cover and subsequently types of bird species are surveyed disproportionately.

Methods

Trend estimation

We compared statewide population trends, based on trend direction and mean annual percent change, between BBS and IMBCR for 148 species across three states: Colorado (2008–2015, N = 132 species), Wyoming (2009–2015, N = 112 species), and Montana (2010–2015, N = 119 species). Length of trend interval varied across states due to variation in the state‐to‐state implementation of IMBCR, which has expanded from Colorado since 2008. We also categorized bird species by their preferred habitat (e.g., forest, grassland, and shrubland; Rodewald 2015) and summarized trend comparisons for those habitat groups.

BBS

We obtained statewide BBS trend estimates readily available from the U.S. Geological Survey Patuxent Migratory Bird Research online trend analysis program (Sauer et al. 2015). BBS trends are provided as an interval‐specific geometric mean, calculated as a ratio of two annual abundance indices and expressed as a percentage change per year with a 95% Bayesian credible interval (Link and Sauer 2002, Sauer and Link 2011).

IMBCR

We applied a state‐space modeling approach to assess population trends using publicly available IMBCR annual abundance data available through the Rocky Mountain Avian Data Center web interface (Bird Conservancy of the Rockies 2018). We chose to use a hierarchal state‐space model to account for both observation error and temporal environmental stochasticity as sources of variability in our trend estimates (Humbert et al. 2009). We modeled trend as the mean instantaneous population growth rate using Markov chain Monte Carlo in a Bayesian framework. By taking one minus the exponentiated instantaneous growth rate we obtained the mean annual percentage change for each species–state trend estimate. Specific details on model structure and fitting can be found in Appendix S1.

Uncertainty and trend

Every resource manager or agency sets their own standards of acceptable levels of uncertainty for a given situation. In situations involving presently abundant or widespread species, managers may wait for reduced levels of uncertainty before initiating additional management actions. For species of concern where any evidence of a decline might prompt action, tolerance of uncertainty may be higher as managers approach conservation decisions more cautiously. We evaluated the uncertainty for each species trend using the posterior distributions from our analysis of IMBCR data, and we used the trend mean and credible intervals provided by the BBS program to draw the BBS posteriors with the assumption BBS trends were normally disturbed. We made this assumption because mean BBS trend values were centered within the credible intervals, which we argue is evidence of a symmetrical distribution.

For each trend estimate, we determined sufficient evidence for a trend occurred if ≥70% of the posterior distribution fell above or below zero. We chose our trend acceptance cutoff of ≥70% support to illustrate a conservation scenario where marginal evidence of a change in population status would warrant further investigation (e.g., a species of concern).

Hypotheses and predictions

Determining whether trends from BBS or IMBCR more accurately reflect truth is a considerable challenge because we do not know the true population changes. However, the characteristics of BBS and IMBCR and subsequent trend analyses provide an opportunity to test specific hypotheses about how trend estimates from these programs differ. We hypothesized the precision of IMBCR trend estimates would be lower than BBS because IMBCR explicitly accounts for the imperfect detection of species in its abundance estimation. We predicted less precise IMBCR trend estimates would yield a larger number of species with insufficient evidence for a trend. For species with nonzero trends, we also predicted the magnitude of the IMBCR trend estimates to be greater than BBS estimates. This occurs because the hypothesized wider posterior distributions (less precise trend estimates) of IMBCR necessitates IMBCR trends to be larger in absolute value to be a nonzero trend. In addition, species may differ in their detectability depending upon the habitats they occupy. For example, tall or dense vegetation can reduce the accuracy of observers during surveying. Since BBS and IMBCR approach the handling of imperfect detection differently, we predicted higher proportion of trend inconsistencies between programs for forest bird species in contrast to species associated with more open habitats such as grassland and shrubland.

Results

Spatial differences in program‐level habitat coverage

We used Landfire existing vegetation type land cover data (Landfire 2014) to categorize the land surveyed by each program into three habitat categories: forest, grassland, and shrubland. Landfire data were sampled within a 400‐m buffer around each route (BBS) and sampling plot (IMBCR). Specifically, we focused on surveys from each program that consisted of a majority for a given land cover category (e.g., ≥50% forest). We mapped the extent of each program across habitats using a one degree latitude by one degree longitude grid, as this is the smallest spatial scale by which BBS is stratified (Bled et al. 2013; Fig. 1). While some overlap between BBS and IMBCR occurs across grassland, forest, and shrubland, IMBCR surveys a broader spatial extent within each state across land cover categories (Fig. 1).

Figure 1

The sampling extent for surveys conducted by each monitoring program over the study period by three land cover types. Plotted are 1° latitude by 1° longitude grid cells that contain surveys with coverage area consisting of a majority for the given land cover category (e.g., surveys with ≥50% of sampled area consisting of forest).

How do trends differ?

We considered trends between programs to differ when species–state comparisons disagreed in direction (increasing, decreasing, or no change). We detected 214 BBS trends and 114 IMBCR trends out of 363 trend comparisons. Overall, 61% (223/363) of the species–state combinations had trend inconsistencies between monitoring programs (Table 1). Most of the differences we observed, 91% (202/223), occurred when one program had a detectable trend while no trend was detected in the other program (Table 1). When trends were detected in both programs, estimates conflicting in direction (e.g., one positive and one negative) 33% of the time (21/63, Table 1). Montana had the largest number of trend differences (70%, 83/119) followed by Colorado (61%, 80/132), and Wyoming (54%, 60/112).

Table 1

Trend comparison summaries for the North American Breeding Bird Survey (BBS) and Integrated Monitoring in Bird Conservation Regions (IMBCR) programs

BBS trend direction	IMBCR trend direction
	Decreasing	No trend	Increasing
Decreasing	16 (4.4%)	58 (16.0%)	12 (3.3%)
No Trend	20 (5.5%)	98 (27.0%)	31 (8.5%)
Increasing	09 (2.5%)	93 (25.6%)	26 (7.2%)

Notes: Results are summarized over all 363 species trend comparisons across Colorado, Montana, and Wyoming, USA. Counts represent the number of species trend comparisons in each trend category. The proportion of trends within each category are shown in parentheses. Boldface type shows instances where monitoring programs had opposite trends for species.

Across all states, species associated with forest habitat had the highest degree of disagreement in trends (61%, 122/199), followed by shrubland (56%, 22/39), and grassland species (56%, 35/62). In Colorado, the largest number of differences were found for forest birds (63%, 46/73) compared to grassland (45%, 9/20) shrubland species (41%, 7/17). In Wyoming shrubland birds exhibited the most trend inconsistencies (60%, 6/10) but inconsistencies were also present at higher degrees for grassland (58%, 11/19) and forest species (51%, 31/61). Of the three states we examined, rates of trend differences in Montana were consistently higher across all habitat types (shrubland 75%, 9/12; forest 69%, 45/65; grassland 65%, 15/23). For a full accounting of population trends for all species–state combinations analyzed see Appendix S2: Table S1.

We also found consistent variation between programs in the estimated size of the proportional changes per year, hereafter trend magnitude. When comparing the change in populations for species with detected trends, we found a unimodal distribution of magnitudes from BBS with most species exhibiting changes to their populations of ±10% per year (Fig. 2). In contrast, the magnitude of trends from IMBCR was bimodal and typically more positive or more negative than BBS trend counterparts (Fig. 2).

Figure 2

Variation in trend magnitudes, represented as proportional change per year, between North American Breeding Bird Survey (BBS) and Integrated Monitoring in Bird Conservation Regions (IMBCR) programs for all species with detected trends.

Discussion

Fundamental program differences drive trend inconsistencies

We found several inconsistencies in statewide trends between BBS and IMBCR. Most trend differences occurred when a trend was detected in one program but not the other and we found more nonzero trends from BBS. Supporting our initial predictions, IMBCR trend estimates were more variable, resulting in fewer trends detected while IMBCR trend estimates were generally larger in magnitude than BBS trends (Fig. 2).

We compared changes in populations over short time periods (at most 8 years) in which stochastic population fluctuations could reduce the probability of detecting a trend. It is also important to note, failing to detect a trend can result from the accurate assessments of stable populations or a lack of information. Nonetheless, when trends were detected in both programs, one in three species comparisons resulted in opposite trends (Table 2). The majority of comparisons in direct conflict (13 of 21) were forest birds and these species had the highest rate of trend inconsistencies overall (compared to grassland and shrubland species). These findings support our prediction that, given that programs differ in their handling of imperfect detection, trends would diverge greatest for species occurring in habitats of more complex vegetation structure.

Table 2

Population trends represented as the mean annual proportional change for species with opposite trends between monitoring programs

Species	Habitat association	State	BBS trend	IMBCR trend
American Redstart (Setophaga ruticilla)	F	MT	−0.03 (0.73)	0.77 (0.71)
Black‐capped Chickadee (Poecile atricapillus)	F	CO	0.03 (0.78)	−0.08 (0.74)
Cassin's Finch (Carpodacus cassinii)	F	CO	−0.03 (0.80)	0.17 (0.83)
Clark's Nutcracker (Nucifraga columbiana)	F	MT	0.07 (0.78)	−0.09 (0.74)
Common Grackle (Quiscalus quiscula)	F	CO	0.01 (0.74)	−0.08 (0.70)
House Finch (Haemorhous mexicanus)	O	CO	0.01 (0.71)	−0.06 (0.71)
House Wren (Troglodytes aedon)	F	MT	0.04 (0.97)	−0.12 (0.75)
Least Flycatcher (Empidonax minimus)	F	WY	−0.07 (0.88)	1.54 (0.79)
Mourning Dove (Zenaida macroura)	F	WY	−0.01 (0.76)	0.13 (0.89)
Northern Rough‐winged Swallow (Stelgidopteryx serripennis)	O	WY	−0.04 (0.74)	1.18 (0.90)
Pine Grosbeak (Pinicola enucleator)	F	CO	0.03 (0.74)	−0.09 (0.78)
Pinyon Jay (Gymnorhinus cyanocephalus)	F	CO	−0.03 (0.85)	0.30 (0.70)
Ring‐Necked Pheasant (Phasianus colchicus)	G	WY	−0.02 (0.78)	0.53 (0.91)
Rock Wren (Salpinctes obsoletus)	O	CO	−0.02 (0.83)	0.09 (0.86)
Savannah Sparrow (Passerculus sandwichensis)	G	CO	0.03 (0.91)	−0.16 (0.76)
Turkey Vulture (Cathartes aura)	O	WY	0.06 (0.98)	−0.12 (0.76)
Warbling Vireo (Vireo gilvus)	F	CO	−0.01 (0.72)	0.09 (0.79)
White‐breasted Nuthatch (Sitta carolinensis)	F	MT	0.06 (0.71)	−0.26 (0.84)
Yellow Warbler (Setophaga petechia)	F	WY	−0.01 (0.85)	0.22 (0.72)

Notes: Trend support, our measure of uncertainty, is calculated as the proportion of the trend posterior distribution that is the same sign as the mean trend value and is listed in parentheses. Species habitat associations are also listed as forest (F), grassland (G), and other (O, which includes species associated with more urban environments or with very narrow habitat requirements). States are MT, Montana; CO, Colorado; and WY, Wyoming.

Although both monitoring programs employ point‐based sampling to monitor bird populations, BBS sampling is constrained to roadside surveys while IMBCR is not (Sauer et al. 2013, Pavlacky et al. 2017). The potential influence of roadside sampling on population estimates has been a critique of BBS. A recent study in Alaska compared short‐term abundance‐based BBS trends (2003–2015) for 31 species to population trends from off‐road surveys and found concordant trends in 58% of the comparisons (25/43; Handel and Sauer 2017). Other research has shown that BBS surveys along roads with more vehicular traffic can be associated with lower bird counts (Griffith et al. 2010), highlighting among other issues the problem of traffic noise reducing an observer's ability to accurately survey. Miscounting birds is an issue for observers of all skill levels. Over‐confident or untrained observers may count more birds than are present (Campbell and Francis 2011), resulting in buffered decline rates or overstated population increases. Equally problematic is the issue of undercounting, where failing to detect present species can lead to exaggerated declines (Thompson 2002). IMBCR explicitly incorporates the detection rates of species on an annual basis when making population estimates (Pavlacky et al. 2017). In contrast, BBS does not model the detection rate for species, though BBS does account for components of the detection process (e.g., observer experience; Sauer et al. 2013). However, we argue without explicitly accounting for imperfect detection, the potential for bias in population estimates remains (Kéry et al. 2009, Nichols et al. 2009, Schmidt et al. 2013).

Before applying data from either BBS or IMBCR, users should be aware that the spatial overlap between these two programs is variable (Fig. 1). The IMBCR program samples across a larger extent of the broad habitat categories we examined: forest, grassland, and shrubland (Fig. 1). Both programs attempt to account for spatial variation in sampling via stratification to inform state‐level population estimates. To calculate a state‐level estimate, IMBCR sums stratum‐level abundance estimates weighted by the survey effort within each stratum (Pavlacky et al. 2017). In contrast, the BBS state‐level composite abundance indices are sums of unweighted stratum indices (Sauer et al. 2013), which we argue may provide an avenue for spatial bias in sampling to persist and influence composite estimates at larger spatial scales. These findings coupled with the potential of detection bias and roadside influences in BBS highlight several plausible explanations for why trend inconsistencies were found.

Conclusions

This is the first study to evaluate population trends using IMBCR products and to our knowledge the first comparison of trends from a large‐scale multi‐state monitoring program to BBS trends. Although we examined aspects of BBS and IMBCR monitoring programs specific to the states of Montana, Wyoming, and Colorado, we posit the lessons from this comparison are transferable to other monitoring programs. For example, recent studies have also found trend inconsistencies between BBS and eBird, an online‐based citizen science program (Walker and Taylor 2017, Horns et al. 2018).

Ultimately, we contend monitoring programs should be developed precisely for the management programs they support (Nichols and Williams 2006). BBS was created with the goal to estimate population trends over long periods of time and this type of data can identify species in need of conservation measures. More research is needed to investigate the influence of detection bias in population trends provided by BBS, especially for use in evaluating short‐term changes to populations. Nevertheless, BBS has considerable value in the platform it delivers for public engagement for bird conservation as well as a half‐century of data providing an avenue to address questions at substantial temporal and spatial scales (Rosenberg et al. 2017).

On the other hand, IMBCR can address conservation objectives across multiple management boundaries by using a robust spatial design that allows for the comparison of annual population estimates across spatial scales (Pavlacky et al. 2017). This type of data is especially useful when identifying how species respond to landscape change via land management practices or natural disturbances like wildfire. Importantly, changes to habitat can influence the abundance as well as the detection rates of species. This underscores the strength of the IMBCR design, which supports the explicit incorporation of imperfect detection across species, management boundaries, and time (Pavlacky et al. 2017). As the longevity of IMBCR increases, the program can also provide long‐term trend estimates. To conclude, our study emphasizes the importance for monitoring programs to be continually evaluated in the context of sampling frame and survey methodology, as well as the analytical methods used. Only through continued refinement and revisiting of methods and analytical approaches, are we able to maintain the reliability of the information used to inform conservation and management.

Click here for additional data file.

Click here for additional data file.