A field experiment characterizing variable detection rates during plant surveys.

Surveys aimed at finding threatened and invasive species can be challenging due to individual rarity and low and variable individual detection rates. Detection rate in plant surveys typically varies due to differences among observers, among the individual plants being surveyed (targets), and across background environments. Interactions among these 3 components may occur but are rarely estimated due to limited replication and control during data collection. We conducted an experiment to investigate sources of variation in detection of 2 Pilosella species that are invasive and sparsely distributed in the Alpine National Park, Australia. These species are superficially similar in appearance to other yellow-flowered plants occurring in this landscape. We controlled the presence and color of flowers on target Pilosella plants and controlled their placement in plots, which were selected for their variation in cover of non-target yellow flowers and dominant vegetation type. Observers mimicked Pilosella surveys in the plots and reported 1 categorical and 4 quantitative indicators of their survey experience level. We applied survival analysis to detection data to model the influence of both controlled and uncontrolled variables on detection rate. Orange- and yellow-flowering Pilosella in grass- and heath-dominated vegetation were detected at a higher rate than nonflowering Pilosella. However, this detection gain diminished as the cover of other co-occurring yellow-flowering species increased. Recent experience with Pilosella surveys improved detection rate. Detection experiments are a direct and accessible means of understanding detection processes and interpreting survey data for threatened and invasive species. Our detection findings have been used for survey planning and can inform progress toward eradication. Interaction of target and background characteristics determined detection rate, which enhanced predictions in the Pilosella eradication program and demonstrated the difficulty of transferring detection findings into untested environments.

INTRODUCTION

When managing threatened or invasive species, failure to detect individuals carries significant consequences. Threatened populations may miss out on crucial conservation actions, leading to further decline and even extirpation (Chadès et al., 2008; Garrard et al., 2015), whereas invasive species may spread and cause further damage (Keith & Spring, 2013; Rout et al., 2014). The most threatened populations and recently arrived invasive species typically have low occupancy rates and may have low detection rates. This makes it difficult to untangle detection rate from focal quantities (e.g., occupancy, abundance, or composition) derived from surveys based on methods of joint inference, such as occupancy modeling (Bornand et al., 2014; Tyre et al., 2003), mark–recapture analysis (Alexander et al., 1997), catch‐per‐unit‐effort modeling (Ramsey et al., 2009), and distance sampling (Robe & Frost, 2002). When individuals are rare or the detection process is variable, experiments that control abundance and directly estimate detection rates can provide stronger inference (Hauser et al., 2015).

Detection experiments set aside estimation of the focal quantity in the short term to estimate detection rate in isolation. They generate replicated and balanced data by controlling and manipulating variation in target type and placement, observers deployed, and aspects of the surrounding environment (Hauser et al., 2012). Live targets, from threatened plants (Alexander et al., 2012) to snakes (Christy et al., 2010), have been used, although target mimics can provide enhanced control and, in the case of invasive species, reduced biosecurity risk. For example, harmless look‐alike plants have been used to represent weeds (Hartley et al., 1989); paint, holes, and tags to represent fungal and insect damage (Bulman et al., 1999; Hauser et al., 2016a; Mangano et al., 2011); and marbles to represent sessile crabs (Delaney & Leung, 2010). For the purpose of more generalized understanding of detection, researchers have painted snail shells to create fictitious pests (Mangano et al., 2011) and deployed coins (McCarthy et al., 2013), gloves, and garbage bags (Robe & Frost, 2002) as targets. Whether the detection rate of these mimics is transferable to real targets is generally unknown, although Pirtle et al. (2021) tested and confirmed their mimic resemblance by using real and simulated leaf damage.

Estimating variation in detection requires substantial replication (“at least a few hundred detection opportunities” [Robe & Frost, 2002]), and detection experiments typically have sufficient power to demonstrate the influence of only a few variables. Most researchers have focused on detection differences arising from variation among targets, for example, differences in species (Chen et al., 2009; McCarthy et al., 2013), size (Bornand et al., 2014; Moore et al., 2011), and flowering (Alexander et al., 2012; Kéry & Gregg, 2003). Detection differences have been established through measurement of uncontrolled variables that describe the surrounding environment, such as dominant vegetation height or density (Bulman et al., 1999; Garrard et al., 2015; Kissa & Shiel, 2012), time of day (Garrard et al., 2008; Hartley et al., 1989), weather (Garrard et al., 2008), and light (Christy et al., 2010; Hartley et al., 1989). The target's surrounding environment is rarely manipulated as part of the experimental design (but see Bulman et al., 1999).

Interactions between variables are likely to influence detection, yet researchers have rarely had sufficient power to test for their effects. In the most thorough investigation of interactions to date, Chen et al. (2009) tested whether shrub and tree detection was influenced by pairwise interactions between target species, observer, survey effort, and target patch size. Among these combinations, they found evidence only of a joint effect of patch size and survey effort. Targets may form a higher visual contrast against some background vegetation types than others (Bohlin et al., 2012), but Chen et al. (2009) did not measure variation in background vegetation.

Interobserver variation in detection has often been found when investigated (Morrison, 2016) and can inform how observers are recruited and trained to maximize detection and inform how survey data are interpreted. In some studies observers with extensive survey experience make fewer mistakes and achieve higher detection rates than observers with less survey experience (e.g., Garrard et al., 2008, 2015; McCarthy et al., 2013; Ringvall et al., 2005), but experienced observers do not perform better in all studies (Alexander et al., 2012; Moore et al., 2011; Morrison & Young, 2016). These inconsistent results may be due to a lack of power (observer experience is rarely directly manipulated) and to difficulty in characterizing relevant experience. Experience tends to be defined qualitatively via an observer's professional identity (Hartley et al., 1989; Mangano et al., 2011; Ringvall et al., 2005) or via a binary judgement that they have performed surveys enough times prior to testing (Alexander et al., 2012; Garrard et al., 2008, 2013; Kéry & Gregg, 2003). It is rare that detection experiments recruit enough participants to test the value of expertise quantitatively (but see Mangano et al., 2011; McCarthy et al., 2013; Ringvall et al., 2005). Kéry and Gregg (2003) and Ringvall et al. (2005) hypothesized that experienced observers’ advantage may be specific to rare species and cryptic forms, but this target–observer interaction remains untested for ecological surveys. Many detection experiments that measure experience report substantial unexplained variation among observers (e.g., Alexander et al., 2012; Garrard et al., 2013; Moore et al., 2011), suggesting that current measures of experience do not accurately describe variation in observer performance.

We were motivated by the detection and eradication of 2 invasive and sparsely distributed Pilosella species in the Australian Alps. Surveys for these targets are undertaken by a variety of observers, from local rangers to community volunteers. Targets potentially vary in their visual contrast against background vegetation, which varies in density and can include flowers of similar color. We conducted a detection experiment in which observers were exposed to potted plants and flower mimics in controlled combinations across stratified background vegetation. We characterized differences in detection rate arising from target type, background conditions, interactions between targets and background vegetation, and 1 categorical and 4 quantitative measures of observer experience.

METHODS

Pilosella species on the Bogong High Plains

Pilosella (previously Hieracium) species are perennial herbs that threaten Australian agriculture and the environment (French, 2021; Williams & Holland, 2007). They are prohibited species at national and state levels in Australia (Agriculture Victoria, 2021; New South Wales Government, 2018). The species occurring in Victoria and New South Wales are subject to an eradication program. Most Victorian infestations occur on the Bogong High Plains in the Alpine National Park and Falls Creek Alpine Resort, where intensive surveys, treatments, and monitoring are conducted. At the time of this study's design, orange hawkweed (Pilosella aurantiaca subsp. aurantiaca [L.] F.W. Schultz & Sch.Bip., previously Hieracium aurantiacum) and king devil hawkweed (Pilosella piloselloides subsp. bauhinia [Schult.] S.Braut. & Greuter, previously Hieracium praealtum) were known to occur in this area.

On the Bogong High Plains, individual plants may flower for 2–3 weeks in summer; peak flowering for the population extends from early December to late January. Surveillance is conducted during this time to maximize the likelihood of successful detection. Hawkweeds are superficially similar in appearance to other non‐native (Taraxacum officinale, Hypochaeris radicata) and native (Picris angustifolia subsp. merxmueller, Microseris lanceolata) yellow‐flowered species occurring in the landscape, although the color of orange hawkweed's flowers is thought to be distinctive.

Surveys to discover hawkweed infestations are conducted by teams of local park staff, state government employees, private contractors, and community volunteers. Teams are assigned to locations of likely hawkweed occurrence, where they form a line with individuals no more than 1 m apart such that each searcher can touch the shoulder of the searchers either side. Individuals search the area in parallel, aiming to thoroughly inspect all of the ground. Research indicates that the optimal survey design is sensitive to hawkweed detection rate (Hauser & McCarthy, 2009; Hauser et al., 2016b). Early attempts to optimally assign search effort relied on detection rates based on expert judgement (Hauser & McCarthy, 2009), which were shown to be optimistic in a subsequent experiment (Moore et al., 2011). However, that experiment investigated only nonflowering hawkweeds and lacked the power to detect an effect of vegetation. We manipulated and tested the influence of the presence and color of hawkweed flowers, cover of surrounding yellow flowers, and surrounding vegetation type on the detection rate of hawkweeds during surveys in the Alpine National Park.

Experimental design

Recreating survey conditions in a controlled experiment poses many challenges (Dennett et al., 2018), and our development of this experimental design was discussed in Hauser et al. (2012). We controlled the placement of hawkweeds in an area that was believed to be free of naturalized hawkweed (<1 km west of Falls Creek Village on the Bogong High Plains). Detection tests occurred in parallel with regular hawkweed surveillance surveys on January 9–13, 2012. Tests were cancelled on January 11 due to bad weather. Our selection of variables to manipulate and measure was guided by hypotheses regarding detection within the survey program and findings from other studies (Table 1).

TABLE 1

Variables measured as potential influences on hawkweed detection rate

Variable	Description	Motivating hypothesis	Type	Observed range a
species b	target species	visual differences (Chen et al., 2009; McCarthy et al., 2013)	categorical	orange hawkweed (baseline), king devil hawkweed
flower b	presence of flower on target	higher color contrast (Kéry & Gregg, 2003; Alexander et al., 2012)	categorical	absent (baseline), present
species × flower b	interaction of target species and flower presence	high contrast limited to orange hawkweed flowers	categorical
plot yellow cover b	cover of background yellow flowers in plot	yellow flowers in background reduce contrast with targets	continuous, standardized	0–0.6%
species × plot yellow cover b	interaction of target species and background yellow flowers in plot	yellow flowers in background change contrast differently for each species	mixed categorical and continuous‐standardized
flower × plot yellow cover b	interaction of target flower presence and background yellow flowers in plot	yellow flowers in background reduce contrast for flowering targets more than nonflowering targets	categorical
species × flower × plot yellow cover b	interaction of target species, target flower presence and background yellow flowers in plot	yellow flowers in background reduce contrast for yellow‐flowering targets more than other targets	categorical
plot vegetation b	dominant vegetation in plot	increasing vegetation density reduces detection (Moore et al., 2011; Garrard et al., 2015)	categorical	grass (baseline), mixed, heath
day	day of search	unexplained day‐to‐day variation in search conditions	categorical	January 9 (baseline), January 10, 12, 13
first day	day of search, first vs. subsequent	observers need time to acclimatize to the task (McCarthy et al., 2013)	categorical	January 10, 12, & 13 (baseline), January 9
time of day, (time of day)2	hours elapsed since 09:00	searcher fatigue reduces detection (Garrard et al., 2008)	continuous, standardized	0.6667–7.3333 h
time elapsed, (time elapsed)2	hours elapsed since observer commenced search day	searcher fatigue reduces detection (Garrard et al. 2008)	continuous, standardized	0–5.9167 h
weather	weather at time of search	observer comfort and lighting increase detection (Garrard et al. 2008)	categorical	sunny (baseline), partly cloudy, cloudy
number visits to plot	the number of times the plot had been visited and searched at the time of this trial	vegetation trampling increases detection (Robe & Frost, 2002; Moore et al., 2011; Alexander et al., 2012)	count, standardized	1–21
ln(number flowers)	number of blooming flowers on stem (log‐transformed)	quantity of color increases contrast and detection	continuous, standardized	1–8 (log‐transformed)
target vegetation height	average vegetation height sampled at 4 corners of 1‐m square quadrat surrounding target	tall vegetation conceals targets and reduces detection	continuous, standardized	4.25–56.63 cm
target yellow cover	cover of background yellow flowers in 1‐m square quadrat surrounding target, measured by counting flowers and measuring a subset of flower diameters (Kendal et al., 2013)	yellow flowers near target reduces contrast with target	continuous, standardized	0–2.5%
species × target yellow cover	interaction of target species and cover of background yellow flowers in 1‐m square quadrat surrounding target	yellow flowers near target affect contrast differently for each species	mixed categorical and continuous‐standardized
flower × target yellow cover	interaction of target flower and cover of background yellow flowers in 1 m square quadrat surrounding target	yellow flowers near target reduce contrast for flowering targets more than nonflowering targets	mixed categorical and continuous‐standardized
species × flower × target yellow cover	interaction of target species, flower and cover of background yellow flowers in 1‐m square quadrat surrounding target	yellow flowers near target reduce contrast for yellow‐flowering targets more than other targets	mixed categorical and continuous‐standardized
role	observer affiliation	observer role affects training, skills, and detection	categorical	Department of Primary Industries (baseline), Parks Victoria, volunteer, contractor, Falls Creek Resort Management
ln(1 + summer hawkweed hours)	number of hours hawkweed search experience the observer had this season (log‐transformed)	recent, specific experience increases detection	continuous, standardized	0–600 (log‐transformed)
ln(1 + summer hawkweed finds)	number of hawkweeds found by the observer this summer (log‐transformed)	recent success increases or predicts higher detection	continuous, standardized	0–140 (log‐transformed)
ln(1 + life hawkweed days)	number of days observer has spent searching for hawkweed in their life	long‐term, specific experience increases detection	continuous, standardized	1–200 (log‐transformed)
ln(1 + life weed days)	number of days observer has spent searching for any weed in their life	long‐term, generalized experience increases detection	continuous, standardized	12–1000 (log‐transformed)
hat	observer use of a hat during search	hats cast shadows and reduce detection	categorical	no (baseline), yes
sunglasses	observer use of sunglasses during search	sunglasses alter color perception and target contrast	categorical	none (baseline), nonpolarized, polarized
colorblind	observer's self‐reported color blindness	color blindness alters color perception and target contrast	categorical	no (baseline), yes

For categorical variables, baseline indicates the level to which an effect size of 0 was ascribed. Continuous variables were standardized by subtracting their mean and dividing by twice their standard deviation.

Variable manipulated during design and included in every model fit.

We manipulated hawkweed target placement to test for differences in detection arising from target species, presence of flowers, and interactions between species and flower presence. Our development of targets balanced a realistic appearance with acceptably low biosecurity risk (Hartley et al., 1989; Moore et al., 2011). We cultivated orange and king devil hawkweeds vegetatively from stoloniferous daughter rosettes of parental plants in a locked glasshouse under permit. Reproductive materials such as stems, buds, and flowers, as well as stolons, were regularly removed and destroyed to prevent hawkweed escape. Pots containing rosettes were buried in the ground with the rim obscured (Figure 1a). To create a low‐risk representation of a flowering hawkweed, we added museum‐quality, varied orange and yellow hawkweed inflorescence mimics crafted from wire and plastic commissioned from Natural History Productions (Darwin) (Figure 1b‐d). We deployed 36 potted orange hawkweeds, 18 of which included orange inflorescence mimics, and 36 potted king devil hawkweed rosettes, 18 with yellow inflorescence mimics.

FIGURE 1

Plants targeted in the survey and observers in the field: (a) a king devil hawkweed rosette (center) in a grass‐dominated plot, (b) a benign king devil hawkweed inflorescence mimic, (c) an orange hawkweed target with an inflorescence mimic (circled) in a plot with high background yellow flower cover, (d) an orange hawkweed target with an inflorescence mimic located on bare ground within a heath‐dominated plot, and (e) participants searching for hawkweeds in 2 separate plots (foreground and background left) while a scribe (background right) records the time to detection of each target found by the paired searcher (background left)

We selected experimental plots to test for differences in detection arising from the density of background vegetation and the density of nontarget yellow flowers. The 16 plots each measured 20 × 20 m, consistent with the resolution of past survey prioritization (Hauser & McCarthy, 2009). Plots were located within walking distance of each other and selected to capture the following vegetation profiles: grass dominated, heath dominated, and mixed grass and heath and low and high abundance of yellow flowers from nontarget species. Cover of background yellow flowers across each plot was estimated from standardized digital photographs of 25 spatially stratified, randomly selected 1 × 1 m quadrats. Kendal et al. (2013) describe how the method was applied to our photographs and demonstrate its efficiency and accuracy as an alternative to manual estimates.

We tested for differences in detection arising from the visual contrast between target plants and background yellow flower cover via 2‐ and 3‐way interactions between target species, flower presence, and the plot‐level estimate of yellow flower cover. An equal allocation of targets to each plot would have maximized statistical discrimination, but we deemed it likely that the consistent pattern would be noticed by searchers and might subsequently influence their search strategy. Instead, we implemented a stratified randomized design, allocating 0, 1, or 2 of each target type to each plot, resulting in 2–7 targets per plot (Table 1 in Hauser et al. [2012]). Targets were placed in the plots at uniform, randomly generated spatial coordinates. Using a higher number of targets would have been possible and yielded more replication, but hawkweeds typically occur at very low densities on the Bogong High Plains, and we hoped to avoid substantial changes in search behavior that might arise from high reward rates (Hartley et al., 1989; Mangano et al., 2011; Moore et al., 2011).

To estimate interobserver variation in detection rate, we arranged for participants to search plots alone instead of in teams and requested that they perform parallel transects across the plots to simulate typical team behavior. We recruited participants from the hawkweed survey teams operating concurrently with the detection experiment, under approval from the Human Ethics Committee at the University of Melbourne (application number 1136572.2). We collected 5 measures of observer experience: professional identity (role in Table 1); number of hours spent searching for hawkweeds this search season (measure of recent, specific experience); number of hawkweeds found by the observer this summer (measure of recent success); total number of days the observer spent searching for hawkweeds in their lifetime (measure of long‐term, specific experience); and number of days observer spent searching for any weeds in their lifetime (a measure of long‐term, generalized experience).

We aimed to schedule observers to minimize time lost to travel and distraction and to minimize systematic biases in search behaviors and data recording. Observers participated in the experiment for up to 1 day each and were partnered with a scribe, who recorded data during searches (Figure 1e). Each observer was assigned a different starting plot corner and a subset of plots that covered all background types in an efficient order of travel. Six to 9 observers participated in parallel each day, allocated to ensure that a variety of plots was visited at any time and observers were unlikely to witness others’ detections. Observers were permitted a fixed time to search each plot, regardless of the perceived difficulty of search at that plot. The fixed time was 30 min on the first day of the experiment. This time was restricted to 20 min on subsequent days when it was decided that briefer searches of more plots were likely to be more informative. Scribes and observers were not informed of the number or type of targets in each plot. Observers marked hawkweeds detected with flags; scribes recorded the time taken to detect the target, species, presence, and number of flowers and approximate location in the plot such that the detection could be later verified by comparison with the known locations of targets. False‐positive detections (i.e., misidentifications of other species as hawkweed) were recorded but excluded from statistical analyses. We measured additional variables (e.g., weather) that we could not control but for which we could test their influence on detection rates (Table 1).

Model fitting and selection

We assumed that the time (t) taken by observer i to detect target k in plot j follows an exponential relationship:

The exponential model is interpreted as random encounters between the observers and targets, and it fits most conveniently into existing models that optimize the allocation of hawkweed search effort (Hauser & McCarthy, 2009; Hauser et al., 2016b). Our experimental data provided observations of t in cases where targets were found. In the cases where targets were present but not found by observers, the detection time was greater than the total time spent searching the plot. Therefore, the data were censored by the total time observer i spent searching plot j, T. Under the exponential assumption, the probability of observer i detecting target k while searching plot j is

We modeled the detection rate λ as a log‐linear function:

where α is the mean log‐detection rate under baseline conditions, X is a vector of fixed effects (Table 1) with corresponding coefficients β and , , , and are random effects for observers, plots, targets, and scribes, respectively. We assumed that the random effects were drawn from 0‐mean normal distributions:

Given the extensive list of measured variables, it was not feasible to compare the fit of every possible combination of variables. We began by fitting a model that included the variables that we controlled: target species, whether the target included flowers, plot‐level dominant vegetation, plot‐level yellow flower coverage, and the 2‐ and 3‐way interactions among target species, target flowers, and nontarget flower coverage. We introduced additional variables to this model 1 at a time in the order they appear in Table 1 (a forward selection algorithm) and omitted a variable if 95% credible intervals on its coefficient included 0. We also noted model deviance information criterion (DIC). We performed all analyses in R (R Core Team, 2019), fitting exponential models with package R2OpenBUGS (Sturtz et al., 2005). Parameter estimates were based on 3 chains each of 100,000 samples thinned to every fifth sample, after a 20,000‐sample burn‐in. We prescribed Normal (0, 106) priors for the variable coefficients and Uniform (0, 100) priors for the standard deviation of random effects. We used pivotal quantities and standardized Pearson residuals to assess the goodness of fit of the preferred exponential model (Conn et al., 2018).

We evaluated the suitability of the exponential model structure by fitting 3 other model structures and comparing them with the exponential model, including only the controlled variables. First, we fitted empirical Kaplan–Meier estimates of the survival function, grouped by the categorical controlled variables (target species, flower presence, background vegetation), with the R package survival (Therneau, 2015). The survival function is equivalent to the probability of detection failure over time, and comparison of the survival functions highlights the circumstances under which the exponential model is a reasonable or a poor fit to the data. A Kaplan–Meier survival function does not support generalized prediction of detection rate and detection probability.

Second, we fitted a Cox proportional hazards model with fixed and random effects with the R package coxme (Therneau, 2018). This model has a more flexible, nonparametric baseline detection rate but fits the same structure of fixed and random effects as the exponential model. It facilitates evaluation of the estimated effects and their consistency, but does not support generalized prediction of detection rate and detection probability.

Third, we fitted a Weibull model with the R package R2OpenBUGS (Sturtz et al., 2005). This model allows for a monotonically increasing, decreasing, or stable baseline detection rate and fits the same structure of fixed and random effects as the exponential model. It allows for comparison of the estimated effects between models and assessment of the exponential model assumption of a stable baseline detection rate. The Weibull model supports generalized prediction of detection rate and detection probability. We used pivotal quantities and standardized Pearson residuals to further compare the goodness of fit of the Weibull and exponential models (Conn et al., 2018). Data and R code for analysis are in Hauser (2021).

RESULTS

We exposed each of the 29 participants to a subset of the 72 hawkweed targets (mean 38.3, range 13–54), creating 1111 opportunities for hawkweed detection. Hawkweeds were correctly detected during 53% of these encounters. Detection was similar between species; 54% of orange hawkweed encounters and 52% of king devil hawkweed encounters led to detection. However, only 25% of nonflowering targets were detected compared with 81% of flowering targets. Sixty‐seven percent of targets were detected in grass‐dominated plots, 45% of targets in mixed grass–heath plots, and 50% of targets in heath‐dominated plots. There were 31 instances of observers misidentifying other species as hawkweeds. These false positives were made by 10 of the 29 observers; 13 of these instances arose from just 1 observer. False positives are of little concern for the hawkweed program because a team leader is available to rapidly correct misidentifications during surveys.

All controlled variables in the exponential model influenced detection rate in at least some circumstances (Figure 2 & Appendix S1). Nonflowering king devil rosettes were found faster than nonflowering orange rosettes, and flowering hawkweeds were found much more rapidly than nonflowering rosettes (Figures 2 & 3). Targets were found more rapidly in grass‐dominated plots than in mixed or heath‐dominated plots. The time required to detect flowering hawkweeds was expected to increase as the density of nontarget yellow flowers in the plot increases (Figure 2), but estimated detection probabilities were more uncertain when nontarget yellow flowers were abundant (Figure 3). Given this uncertainty, we did not discern changes in the detection rate for nonflowering hawkweed rosettes as a response to nontarget yellow flower abundance.

FIGURE 2

Coefficients of standardized variables in the preferred plant‐detection model (dots, posterior means; lines, 95% credible intervals; positive coefficients, targets detected more rapidly than under the baseline case; negative coefficients, targets detected more slowly; cf., variable level compared with baseline level [e.g., effect of king devil hawkweed species compared with the baseline orange hawkweed species]; *, variable manipulated in the experiment). The model was fit to detection times measured in minutes and the detection area of 20 × 20 m

FIGURE 3

Detection‐effort curves for a search of a 20‐ × 20‐m plot given the target is flowering orange hawkweed (red), nonflowering orange hawkweed (orange), flowering king devil hawkweed (purple), or nonflowering king devil hawkweed (blue): (a, d, g, j) grass‐dominated plots, (b, e, h, k) mixed‐grass and heath plots, and (c, f, i, l) heath‐dominated plots that contain (a–f) 0.0% or (g–l) 0.6% cover of nontarget yellow flowers (i.e., the highest cover observed in the study) (lines, median detection; shading, 95% credible intervals without random effects). The assumptions are that plots have not been visited previously, search occurs at the median time of day (13:10), observers spend the median time (20 h) searching for hawkweeds in that summer, and there are no nontarget yellow flowers in a 1‐ × 1‐m quadrat around the target

The preferred model additionally included 4 of the recorded covariates that were not experimentally controlled (Figure 2 & Appendix S1). Time of day had a quadratic effect on detection rate: detection improved for the first 4 h after 09:00 and declined thereafter (Figure 4a). Time to detection was estimated to decrease slightly the more often a plot had been visited (Figure 4b). Nontarget yellow flower cover occurring close to the target was estimated to slow detection, but was highly uncertain (Figure 4c). Participants who had more experience searching for hawkweeds that same summer detected hawkweeds slightly more rapidly than other observers (Figure 4d). Compared with a median observer with 20 h experience, the estimated detection time increased by 37% for an observer with 0 h experience and decreased by 26% for an observer with 600 h experience.

FIGURE 4

Influence of (a) time of day, (b) number of visits previously made to the plot, (c) the cover of nontarget yellow flowers in a 1‐m quadrat around the target, and (d) number of hours searcher spent seeking hawkweed in that season expressed as a multiplier on the detection time (purple, median and 95% credible intervals; green circles, observations). All 4 variables were standardized; thus, the multiplier is 1 when the variable is at its mean value in the data set

There was substantial between‐target variation (Figure 2), such that detection times for 95% of targets could be as low as one‐third and as high as 3 times the average detection time. After accounting for observers’ recent hawkweed search experience, there remained substantial unexplained between‐observer variation. The poorest performing 5% of observers were estimated to have at least 37% longer detection times than the average observer, and the best‐performing 5% of observers were estimated to have at least 27% shorter detection times than the average observer. Between‐plot variation was similar in magnitude to between‐observer variation, and between‐scribe variation was slightly lower. In combination, the estimated random effects contributed substantial variation around the estimated probability of detecting hawkweed (Appendix S2).

During the selection of uncontrolled predictor variables, coefficient estimates remained relatively stable (Appendix S1). Including the 4 recorded covariates in the preferred model reduced DIC to 4437 from the baseline model DIC of 4441, and goodness‐of‐fit tests indicated that the baseline and preferred models were of similar quality with no systematic patterns in residuals (Appendix S3). The observed data included many more long detection times than would be expected from exponentially distributed data, and this was investigated further via comparison with other model structures.

Comparisons among the exponential, Weibull, and Kaplan–Meier survival functions confirmed that the exponential model adequately captured changes in detection probability explained by the controlled variables and that the additional complexity of the Weibull model did not offer improved fit (Appendix S4). Comparisons of coefficients among the Cox proportional hazard, exponential, and Weibull distributions confirmed that the relative influence of each variable was consistent across model structures. The key weakness of the preferred exponential detection model was that it estimated faster detection times for flowering targets in grass‐dominated plots than was observed in the data set. The Weibull distribution allows for a heavier‐tailed distribution of detection times (when 0 < v < 1) but this did not emerge as the best‐fitting Weibull model (v = 1.1).

DISCUSSION

Our detection experiment quantified how target and background characteristics can interact to determine detection rate in a visual plant survey, which has not been achieved in other detection experiments. This was enabled by a high replication of search encounters across the experiment, stratification across the environmental background, and our control of the number and placement of nonflowering, yellow‐flowering, and orange‐flowering hawkweed targets. When the distribution of interacting target and background characteristics can be predicted across the survey landscape, such interactions can be incorporated into survey design.

The detection model we developed is compatible with established methods for survey design, which rely on estimates of detection probability (Chadès et al., 2011; Regan et al., 2006) or detection rate (Garrard et al., 2008; Rout et al., 2014). The specific estimates from this study are applied during annual hawkweed survey planning (Constantine et al., 2016; Hauser et al., 2016b). The effect of background vegetation on detection rate is used in combination with maps of dominant vegetation across the survey landscape to optimize the spatial allocation of effort during preseason planning. Plans could also partially account for differences in detection between species and between flowering and nonflowering targets that we quantified in this study because the survey season extends beyond (but focuses on) hawkweeds’ brief flowering season (Bonneau et al., 2018) and the species’ populations have distinct but overlapping spatial extents. However, the prevalence of nontarget yellow flowers is unknown across the landscape. Although it cannot be accommodated during the annual survey design, survey team leaders could be trained to visually assess coverage of yellow flowers at the commencement of a survey and adjust the team's survey effort to meet the detection standard set in the annual survey design. Survey effort could also be adjusted throughout the day to account for the time‐of‐day effect estimated in our detection model. Even if survey effort cannot be adjusted in the field, data recorded during the survey regarding background yellow flower coverage or time of day could be used to support data analyses, especially when no hawkweeds are found.

Our measurement across 3 different vegetation types (grass dominated, mixed grass and heath, and heath dominated) did not generate the monotonic decrease in detection rate that we anticipated. Detection in grass was easiest, as expected, but hawkweeds were just as difficult to find in medium‐density mixed grass–heath vegetation as in higher density heath‐dominated vegetation. Observers were free to choose their search speed and focus, and it may be that their chosen allocation of effort between grass and heath within a mixed plot was not optimally efficient, such that they allocated too much time to inspecting grass, where it was easy to detect targets, and too little time inspecting heath, where it was difficult to detect targets. Another possibility is that additional search effort is needed to address the increasing visual complexity.

We also made progress in characterizing the influence of observer experience on detection rate due to our use of 1 categorical and 4 quantitative measures of observer experience. Although others tended to focus on longer term measures of experience (e.g., McCarthy et al., 2013; Ringvall et al., 2005), we found evidence that the observers’ detection rate slightly improved with recent, specific experience (i.e., the number of hours spent searching for hawkweeds within that season). Variables relating to professional affiliation, recent detection success prior to the experiment, and long‐term experience did not influence performance in the detection experiment. It is encouraging that community volunteers demonstrated the potential to detect hawkweeds as effectively as professional staff because many conservation programs include a large volunteer effort (Benshemesh et al., 2018; Cherry et al., 2016). Our community volunteers were not an unbiased sample of the general public, and many reported other forms of relevant experience in the questionnaires, including professions and tertiary qualifications associated with environmental or agricultural work. Thus, many volunteers may hold relevant long‐term experience that is not captured in standard assessments.

The detection rates we estimated were lower than comparable previous estimates for orange hawkweeds in Victoria, which were derived from expert opinion (Hauser & McCarthy, 2009) and a field experiment with lower sample size (Moore et al., 2011). Nevertheless, we expect that our preferred model may overestimate detection rates for 3 main reasons. First, observers were exposed to a substantially higher quantity of targets than in a typical survey, where it is possible to search for days without detecting a new infestation, and the higher reward rate may have boosted their motivation (Hartley et al., 1989; Mangano et al., 2011). Second, observers’ awareness that they were being assessed may have increased their vigilance (Moore et al., 2011). Third, comparison of empirical detection functions and the preferred detection model (Appendix S4) suggest that the realized pattern of detections may include slower detection of flowering hawkweeds and a higher frequency of very‐difficult‐to‐detect hawkweeds. An optimal allocation of surveillance effort assuming lower detection rates is likely to prescribe a higher search effort per site across fewer sites and include fewer hard‐to‐search (i.e., mixed and heath‐dominated) sites (Hauser & McCarthy, 2009).

Despite the high replication in our data set and the meaningful effects characterized in our detection model, random effect estimates indicated that unexplained variation remains, particularly among targets. This could potentially be accommodated during survey design. Survey planning typically focuses on the predicted average detection, but if a program aims to achieve eradication through visual detection of every infestation, then it may be more useful to develop search strategies with detection rate estimates for the hardest‐to‐detect infestations. Depending on the managers’ attitude to risk, a robust‐optimal survey allocation could spread effort more evenly across sites or intensify effort at high‐risk sites, relative to methods that focus on average outcomes (McCarthy et al., 2010; Yemshanov et al., 2019).

In the pursuit of hawkweed eradication, our findings motivated managers to explore a broader range of survey tools to complement visual searches, including algorithmic detection of hawkweed flowers from drone imagery and olfactory detection by dogs (Cherry et al., 2016). For other invasive and threatened species programs that rely on visual searches, our findings demonstrate the influence of target‐background interactions on detection and the potential for high variability among targets and observers. This limits the transferability of our and other studies’ detection rates, and research that helps generalize our understanding and estimates of detection will be highly valuable. Progress could be made via trait‐based models of detection across multiple species and backgrounds (e.g., Garrard et al., 2013), quantitative measures of visual contrast and complexity (e.g., earth mover's distance; Kendal et al., 2013), and pre‐survey assessment of observers’ aptitude for detection.

Detection experiments are a direct and accessible means of evaluating the strengths and weaknesses of visual surveys. We quantified what hawkweed managers, observers, and researchers presumed regarding the relative ease of detecting orange‐flowering, yellow‐flowering, and nonflowering targets. Our results contradicted past assumptions that hawkweeds are easier to find in mixed vegetation than heath‐dominated vegetation and that most nonflowering hawkweeds could feasibly be detected under standard survey protocols. More specifically, our study provided quantitative estimates that have been used in survey planning and data analysis for 8 years (Constantine et al., 2016; Hauser et al., 2016b) and can inform progress toward hawkweed eradication. Our method of high replication and balanced design allows for the investigation of many controlled and uncontrolled variables to form a more nuanced understanding of the detection process and subsequently more effective survey designs for rare species.

Appendix S1. Accessible via https://figshare.com/s/4aa290ea79bf0b1f2ea5 as a spreadsheet.

Appendix S2. Detection‐effort curves including random effects

Figure S2.1. Detection‐effort curves for a variety of scenarios in a 20m x 20m plot search, given the target is orange hawkweed with (red) or without (orange) a flower.

Figure S2.2. Detection‐effort curves for a variety of scenarios in a 20m x 20m plot search, given the target is king devil hawkweed with (purple) or without (blue) a flower.

Appendix S3. Goodness‐of‐fit testing for the preferred model

Figure S3.1. Comparison of (a) cumulative distribution functions and (b) quantiles between the EXP(1) distribution and the transformed selected model.

Figure S3.2. Histogram of residuals for the selected model.

Figure S3.3. Plots of residuals against the linear predictor, and each of the variables in the preferred model.

Appendix S4. Fitting and comparing four model structures

Figure S4.1. Estimated survival functions for detection data.

Figure S4.2. Parameter estimates arising from the Cox proportional hazards, exponential and Weibull model structures (mean and 95% confidence/credible intervals).

Figure S4.3. Comparison of cumulative distribution functions against the EXP(1) distribution for the exponential (left) and Weibull (right) detection models (further description of this approach appears in Appendix B).

Figure S4.4. Comparison of distribution quantiles against the EXP(1) distribution for the exponential and Weibull detection models (further description of this approach appears in Appendix B).

Figure S4.5. Histograms of residuals for the exponential and Weibull detection models.

Figure S4.6. Residuals plotted against predictors for the exponential and Weibull detection models.

Click here for additional data file.