Economic assessments from experimental research trials of feedlot cattle health and performance: a scoping review.

Animal husbandry decisions for feedlot cattle may be based on economic or financial impacts reported from livestock research trials comparing interventions such as health practices or performance technologies. Despite the importance of economic assessments to production management decisions, there are no consensus guidelines for their methods or reporting. Thus, we hypothesized that methods and reporting of economic assessments in the scientific literature are inconsistent. This scoping review describes the types of economic assessments used to evaluate the costs and benefits of interventions in feedlot trials, how measured health and performance outcomes are utilized in economic evaluations, and the completeness of reporting. A structured search was used to retrieve peer-reviewed articles (published in English) on experimental trials performed in Australia, North America, or South Africa, which reported feedlot cattle health, performance, or carcass characteristics and included an economic outcome. A total of 7,086 articles were screened for eligibility; 91 articles (comprising 113 trials) met the inclusion criteria. Trial characteristics, methods, and reporting data were extracted. A primary outcome was stated in only 36% (41/113) of the trials. Of these 41 trials, an economic outcome was reported as a primary outcome in 18 (44%). Methodology for the economic assessment was reported for 54 trials (48%), the type of economic assessment was explicitly stated for 21 trials (19%), and both the type of economic assessment and methodology used were reported for 29 trials (26%); neither were reported for nine trials (8%). Eight types of economic assessments were explicitly reported: cost-effectiveness, cost-benefit analysis, enterprise analysis, partial budget, break-even analysis, profitability, decision analysis, and economic advantage. From the trials that did not report an assessment type, three were identified: partial budget, enterprise analysis, and gross margin analysis. Overall, only 32 trials (28%) reported economics as an outcome of interest, the methodology used or the type of assessment, and values, sources, and dates for at least some of the price data used in the analysis. Given the variability in methods and inconsistent reporting for feedlot trials identified by this scoping review, a guideline to facilitate consistency on appropriate methods and reporting is warranted.

INTRODUCTION

Experimental research, such as randomized field (clinical) trials, provide strong evidence of causation and are especially suited to evaluate the efficacy or effectiveness of interventions that aim to improve animal health, performance, food safety, and other outcomes of importance to livestock production systems (Sargeant et al., 2010). Economic or financial outcomes are often important drivers of management decisions regarding the adoption or implementation of interventions within a production system. Within feedlot production systems, an assessment of economic outcomes is often the primary goal of a field trial in which two or more health or production management strategies are compared. Although appropriate inferences or management decisions based on results of feedlot trials require valid research and transparent reporting (Sargeant et al., 2010), there are no established guidelines regarding economic or financial assessments to ensure the research methods and results are performed and reported accurately.

Guidelines for reporting economic assessments in human health studies, such as the Consolidated Health Economic Evaluation Reporting Standards (CHEERS; Husereau et al., 2013), have been established. In addition, there have been a variety of resources and guidelines established for reporting research on animals, such as the reporting guidelines for randomized controlled trials for livestock and food safety (REFLECT; O’Connor et al., 2010; Sargeant et al., 2010), with several of these resources currently provided at the MEnagerie of Reporting guIDelines Involving Animals (MERIDIAN) website (https://meridian.cvm.iastate.edu/). In general, these guidelines aim to improve the reproducibility and reporting of research studies so that the results are more useful for stakeholders. However, neither CHEERS, due to the often differing objectives for economic analyses for human health compared to livestock production (Babo Martins and Rushton, 2014; Sanghera et al., 2015), nor any of the established guidelines or standardized approaches for animal research directly facilitate the accurate reporting of research methods and results when economic or financial assessments are a critical component of the livestock field trial.

Given the practical importance of economic assessments in livestock field trials, yet the gap in the literature regarding guidelines for this research area, we hypothesized that numerous publications would exist, but the reporting of key components would not be consistent. Since no previous research has assessed the reporting of feedlot trials with economic results, we performed a scoping review, which is a systematic and transparent approach to chart and assess published literature on a defined topic (Levac et al., 2010; Tricco et al., 2018). The two primary objectives of this scoping review were to chart: 1) the types of economic assessments used to evaluate the costs and benefits of interventions on beef cattle health and performance metrics measured in a feedlot or corresponding carcass characteristics, from experimental trials, and 2) how these measured health and performance outcomes are utilized in the trials’ economic assessments. The secondary objective was to summarize how the economic assessment methodology was reported regarding the reproducibility of the analysis.

MATERIALS AND METHODS

This scoping review was carried out utilizing the framework of Arksey and O’Malley (2005), with the recommendations of Levac et al. (2010), except no stakeholder consultations were performed. A protocol for this scoping review was developed a priori by group discussion among coauthors. This protocol, along with recorded alterations and rationales, is presented in Supplementary Material 1.

Article Sources

In April 2020, articles were identified through a search of four electronic databases: CAB Direct (1930 to 2020), Web of Science – CORE collections (1945 to 2020), Scopus (1800s to 2020), and Pubmed (1950 to –2020), using the database’s default time frame, through the Kansas State University library. The CAB Direct and Scopus databases were chosen as they give the most comprehensive coverage of important veterinary medicine journals and we included Web of Science and PubMed to be as comprehensive as possible in our search (Grindlay et al., 2012). Citation information and abstracts from the database searches were exported into a spreadsheet program (Microsoft Excel), with each article assigned a unique identification number. Duplicates were manually removed. In addition to the database search, any relevant systematic reviews or meta-analyses that were identified from the search had their references hand-searched for relevant articles that fit the eligibility criteria; however, only original research was included in the scoping review. Any relevant articles from the authors’ personal collections that were not previously identified by the database search nor from a review or meta-analysis evaluation also were included.

Eligibility Criteria

Articles were eligible for inclusion if they had been published in a peer-reviewed journal and written in English. Our population of interest was beef cattle housed in feedlot operations in Australia, North America (Canada, United States of America, and Mexico), or South Africa. We defined beef cattle as non-dairy breed cattle raised exclusively for beef. Dual-purpose breeds were included only if it was conclusive that they had been raised for beef consumption and not milk production. We defined a “feedlot” based on the Environmental Protection Agency’s (EPA) definition of an animal feeding operation (EPA, n.d): animals needed to be, or would have been if not for the length of the trial in place, confined and fed for 45 d or more and pre-harvest crops raised at the facility were not used as the primary diet source for cattle. We limited our study locations to the primary areas of large-scale commercial feedlot production in order to ascertain that animals were under management representative of large-scale commercial beef feedlots. The beef industries in these areas utilize a finishing phase where cattle, in operations generally housing more than 1,000 animals, are fed high-grain diets in densely confined non-pastured lots (United States Department of Agriculture, n.d; Australian Lot Feeders Association, n.d; South African Feedlot Association, n.d; Statistics Canada, n.d).

Articles were eligible if they included results from experimental trials, performed at either a research or commercial feedlot facility, and a health and/or performance metric measured in a feedlot and/or carcass characteristic was measured as the outcome of interest(s). Additionally, articles reporting an economic/financial outcome, defined as a monetary cost or benefit from an assessment of an intervention on feedlot outcomes, were deemed eligible.

Search Terms

The question of interest was: what types of economic/financial assessments have been used to evaluate the costs and benefits of interventions on beef cattle health and performance metrics from experimental field trials? The search strategy was developed to target articles that included experimental studies AND beef cattle AND economic analysis, as search terms. The exact search terms were developed through discussion with the coauthors defining the eligibility criteria and then using an iterative strategy, where a set of terms were identified and the returned titles were assessed for broad compliance and the inclusion of known articles that fit the eligibility criteria. The terms were then updated and the search re-deployed assessing the returned titles until all known articles were found and there was little improvement in title relevance. Search terms for the four database searches are reported in Table 1.

Table 1.

Databases and the respective search strategies used in this scoping review

Database	Interface	Dates included1	Search terms
CAB Direct2	CAB Direct	1930–2020	All fields = ((cattle OR “cow-calf” OR “feed lot” OR “feed lots” OR feedlot* OR ((calves OR calf) NOT human) OR cow* OR steers OR bull OR beef) NOT (“cowpea” OR lamb OR goat OR dairy OR buffalo OR camel)) AND (“field trial” OR “clinical trial” OR “experimental trial” OR randomized OR randomization OR trial) AND (“economic analysis” OR “financial analysis” OR economic* OR cost OR price OR “willingness to pay” OR benefit OR minimization OR effectiveness OR consequence)
Web of Science – CORE collections	Web of Science	1945–2020	TS=((cattle OR “cow-calf” OR “feed lot” OR “feed lots” OR feedlot* OR ((calves OR calf) NOT human) OR cow* OR steers OR bull OR beef) NOT (“cowpea” OR lamb OR goat OR dairy OR buffalo OR camel)) AND (TS=(“economic analysis” OR “financial analysis” OR economic*) AND (cost OR price OR “willingness to pay” OR benefit OR minimization OR effectiveness OR consequence)) AND TS=(“field trial” OR “clinical trial” OR “experimental trial” OR randomized OR randomization OR trial) AND LANGUAGE: (English)
Scopus	Elsevier	1800s–2020	TITLE-ABS-KEY((cattle OR “cow-calf” OR “feed lot” OR “feed lots” OR feedlot* OR ((calves OR calf) AND NOT human) OR cow* OR steers OR bull OR beef) AND NOT (“cowpea” OR lamb OR goat OR dairy OR buffalo OR camel)) AND (“field trial” OR “clinical trial” OR “experimental trial” OR randomized OR randomization OR trial) AND (“economic analysis” OR “financial analysis” OR economic* OR cost OR price OR “willingness to pay” OR benefit OR minimization OR effectiveness OR consequence) AND LANGUAGE(English) AND AFFILCOUNTRY(“United States of America” OR australia OR “South Africa” OR mexico OR canada)
Pubmed	NCBI	1950–2020	All fields ((cattle OR “cow-calf” OR “feed lot” OR “feed lots” OR feedlot* OR ((calves OR calf) NOT human) OR cow OR cows OR steers OR bull OR beef) NOT (“cowpea” OR lamb OR goat OR dairy OR buffalo OR camel)) AND (“field trial” OR “clinical trial” OR “experimental trial” OR randomized OR randomization OR trial) AND (“economic analysis” OR “financial analysis” OR economic* OR cost OR price OR “willingness to pay” OR benefit OR minimization OR effectiveness OR consequence) AND ENGLISH[LANGUAGE]

TS = Topic; searches abstract, author keywords, and Keywords Plus.

Searches were not limited by date and the range presented is inclusive of each database’s default timeframe searched. The search took place April 2020.

Includes CAB Abstracts, Global Health, VetMed Resource, CABI Full Text, Distribution Maps Plant Diseases, CAB eBooks Archive 2008–2010, Animal Health and Production Compendium (AHPC), eBook Choice Kansas State Collection.

Relevance Screening

Two rounds of relevance screening were performed using a list of questions, created using our eligibility criteria and refined through group discussion with coauthors, to guide the screening process (Supplementary Material 1, section “Selection process”). In the first round, two reviewers independently evaluated the title and abstract of all articles. The full-texts from articles included after the first round were retrieved. In the second round, the full-text of each article was also screened in duplicate. Disagreements between the two reviewers’ answers were resolved through discussion and if a consensus could not be reached a third reviewer was consulted.

Data Extraction Process

Based on group discussion between coauthors, we created and refined the data extraction form, which defined the data items to be extracted. The final data extraction form, approved responses, alterations, and rationales are available from authors upon request. Two reviewers independently extracted data items from relevant full-text articles into a spreadsheet program (Microsoft Excel). If an article contained information regarding more than one experimental trial, data were extracted for each trial individually. If there was disagreement between the reviewers in terms of data extracted, a consensus was reached using the same process described for the screening process. Authors of articles were not contacted to obtain any additional information.

Extracted Data Items and Explanations

A summary of the data items extracted is presented in Table 2 and detailed explanations are given in this section. Except for citation information, all data items were extracted at the trial level and a single article could include information from multiple trials.

Table 2.

Summary of the key data items and their description, by the level at which they were extracted

Article level	Description
Citation information	The name of the first author, a list of all authors, title, journal name, and publication date

Location and animal management items

Information about the country where the trial took place and the U.S. state or province/territory, for non-U.S. countries, was extracted as reported in the article or if only a city name was reported, the reviewers confirmed the country and state/province/territory where the city is located. If a trial took place in more than one state/province/territory, it was marked as “multiple.” If geographic location and/or type of facility where the trial took place was not reported in the text, the reviewers first attempted to extract the information from the article authors’ affiliation(s) and/or acknowledgement. If this was ambiguous, the following decisions were made: the reviewers extracted the country and state/province/territory of the article authors’ affiliated institution for these respective items, if there was more than one state/province/territory represented by the article authors’ affiliated institutions then the state/province item was reported as missing. If the facility type where the trial took place could not be determined from the affiliation and/or acknowledgement then it was reported as missing.

For the pen/paddock size, if a capacity range for each pen/paddock was reported, this information was extracted, unless the actual number of animals allotted to each pen/paddock was higher than the given capacity, then the exact number allocated was extracted for this item. If only the number of animals allocated to each pen/paddock was reported, this datum was extracted.

Intervention and study design items

For the total number of animals in the trial, reviewers extracted all applicable information that was reported. If the total number of animals was not reported, reviewers marked this item as missing unless it was possible to unambiguously calculate the total (e.g., 30 animals in treatment A and 30 animals in treatment B for 60 total animals). Reviewers extracted the production stage when the intervention was applied using the following criteria to differentiate between stages: interventions applied to the dam and/or to the calf pre-weaning were deemed as “cow-calf,” weaning management or interventions applied to young/light weight cattle in drylots or on pasture before the finishing stage were defined as “backgrounder or stocker,” and if the intervention was applied to group-penned cattle at the finishing stage, including interventions applied during processing such as prophylaxis and metaphylaxis, were classified as “feedlot.” If multiple interventions were applied at different production stages, information from all stages was extracted.

The reviewers recorded the intervention applied in the trial as reported by the authors and created the following categories based on the extracted interventions: vaccination, parenteral antimicrobial, parasite control, feeding, management, and multiple interventions. Vaccination interventions included comparisons of a vaccine to a control or to another product, as well as the timing of vaccine administration. Parenteral antimicrobial interventions included comparisons of injectable antimicrobials used for metaphylaxis, disease treatment, or both. Parasite control interventions included comparisons between de-worming products as well as the effect of the magnitude of parasite burden. Feeding interventions included the following: comparisons among diets, feedstuffs, or feeding strategies, the effects of feed additives (e.g., beta-agonists, ionophores, melengestrol acetate [MGA], and antimicrobials) or supplements, or a combination of these feeding interventions. Management interventions included investigation of the effects of pre-conditioning systems, feedlot management systems, abortion of feedlot heifers, sorting, shade, administration of implants, and exposure to cattle that are persistently infected with bovine viral diarrhea virus. If interventions of multiple categories were applied, the trial intervention was categorized as “multiple.”

Outcomes were categorized into one of four categories: carcass (e.g., hot carcass weight, yield and quality grades), economic (i.e., any financial or economic objective), health (e.g., morbidity, mortality rate, antibody titers), and live performance (e.g., average daily gain, dry matter intake, body weight). We used these broad groups for ease of reporting, as our objective was to chart how health and performance outcomes are used in economic/financial assessments not which specific health/performance outcomes are used. If primary and secondary outcomes were stated (“yes”), the outcome category was further defined as “primary outcomes,” “secondary outcomes,” and/or “outcomes that were measured, but not reported in the objectives” (i.e., any outcome that was reported in the text, but not referenced in the objective statement). If primary and secondary outcomes were not stated (“no”), the outcomes were further defined as the “outcome(s) of interest” and/or “outcomes that were measured, but not reported in the objectives.” A single trial could contain multiple types of outcomes (e.g., “primary outcome” and “secondary outcome”) and these could be from the same category (e.g., health).

Economic/financial assessment items

The reviewers extracted if the values used in the economic/financial assessment were reported and if the source or methods to calculate that value were reported; categorizing the source/methodology as a “public,” “private,” or “commercial service.” Public sources included any report or publication that is in the public sphere (e.g., journal articles, government reports), private sources were values unique to the trial, facility, and/or producer (e.g., feedlot receipts, producer specific carcass grids), and commercial services were sources that are available to paying members of an association or group (e.g., Canfax). The reviewers also extracted if the dates the values were sourced/estimated were reported (“yes”/“no”). This received a “yes” if it was stated, but also if the article authors reported the values were privately or commercially sourced and the date of the trial was also reported; otherwise, it received a “no.” The reviewers used text, tables, and footnotes to assess questions pertaining to values and sources.

RESULTS AND DISCUSSION

Article Retrieval

A total of 8,244 articles were identified from the four database searches and 1,315 duplicates were removed. From the unique articles (n = 6,929), 138 eligible reviews were identified; 18 reviews were not searched—10 were specific to noneligible locations and 8 could not be retrieved. The reference lists of 120 reviews were searched by hand and 158 new articles were identified. One article could not be accessed; therefore 157 articles’ titles and abstracts were added (“Relevance Screening: Title and abstracts” phase; Figure 1). From the 7,086 titles and abstracts that were screened, 184 articles were deemed relevant. During the full-text screen, three additional reviews were identified and their reference lists hand-searched. One relevant article was identified and included in the full-text screen of 191 articles (“Relevance Screening: Full-text”; Figure 1). From the full-text screen, 91 articles met the eligibility criteria and were charted, whereas 100 articles were removed; reasons for removal are listed in Figure 1.

Figure 1.

A flow chart of the number of articles identified, screened for eligibility through two rounds of relevance screening, a title and abstract screen and a full-text screen, and the final number of articles retained for data extraction. The number of articles excluded by the eligibility screening with reasons are detailed.

Characteristics of Articles

From the 91 articles that were charted, the median year of publication was 2005 and ranged from 1967–2018 (see Supplementary Figure 1 in Supplementary Material 2). Most of the articles were published in 2008 (n = 13). Nineteen journals were represented in the scoping review, and over half of the articles (59%) were published in three journals: The Professional Animal Scientist (22%; n = 20), Journal of Animal Science (21%; n = 19), and The Canadian Veterinary Journal (16%; n = 15) (see Supplementary Table 1 in Supplementary Material 2). Although it limits the scope of this review, including peer-reviewed articles only, as opposed to also including “grey literature,” increases the rigor of the reported results and minimizes bias. Efforts were made to include not only peer-reviewed articles retrieved from databases encompassing agricultural research but also via hand search of relevant review articles.

Characteristics of Trial Populations

One hundred and thirteen eligible trials were identified from the 91 articles. Sixty-seven trials (59%) were performed in the United States, 42 (37%) in Canada, and four (4%) in South Africa. Limiting the geographical scope of the review and excluding regions with growing feedlot industries, such as Brazil, Argentina, or Uruguay, resulted in the exclusion of potential eligible articles. However, the aim of this review was to describe trials from geographic regions with large commercial feedlot industries traditionally similar to those found in the United States. Historically, Brazil has been largely pastoral, with intensification of the beef industry occurring predominately in the last couple of decades (Millen et al., 2011). Lastly, and related to the geographical location, the exclusion of non-English publications further limits the scope of our findings.

Fifty percent of trials (57/113) were performed at a commercial facility, 37% (42/113) at a research facility, and the type of facility used was not reported for 12% (14/113) of trials. Overall, the median number of animals per trial was 361 and ranged from 22 to 19,099. Commercial facilities performed larger trials with a median trial size of 1,010 animals (range = 103 to 19,099), whereas trials performed at research facilities had a median trial size of 215 animals (range = 22 to 2,659). Trials at unspecified facility types had a median trial size of 143 animals (range = 64 to 774). Sixty-six percent of commercial trials (38/57) used pen sizes that held 51 to 350 animals, whereas 43% (18/42) of research trials and 50% (7/14) of trials at unspecified facilities used pen sizes that held up to 15 animals. Overall, pen sizes of 51 to 350 animals were used in 36% (41/113) of trials, the number of animals allocated per pen was not specified in 26% (30/113), and pen sizes of up to 15 animals were used in 23% (26/113) of the trials. Interventions were applied at the feedlot stage in 80% (90/113) of the trials, whereas interventions were applied at the background or stocker stage, at the cow-calf stage, or at multiple stages in the other 20% (23/113) of the trials. Trials by production stage at intervention, type of intervention, and pen/paddock size are presented in Table 3.

Table 3.

The number of trials (n = 113) by production stage at intervention, type of intervention, and housing group size with citations

Production stage at intervention	Intervention	Pen/paddock size	N	Citations
Cow-calf (n = 7)	Feeding	≤ 15	1	Larson et al., 2009
		Unspecified	2	Mulliniks et al., 2012; Stalker et al., 2006
	Management	Unspecified	3	Karren et al., 1987; Peterson et al., 1989
	Vaccination	Unspecified	1	Kirkpatrick et al., 2008
Cow-calf and backgrounder or stocker (n = 2)	Management	Unspecified	1	Shike et al., 2007
	Multiple	16–50	1	Anderson et al., 2005
Cow-calf and Feedlot (n = 1)	Vaccination	Unspecified	1	Bechtol et al., 1991
Backgrounder or stocker (n = 8)	Feeding	≤ 15	2	Forte et al., 2018; Mir et al., 2008
		Unspecified	3	Parish et al., 2013; Lewis et al., 1990; Bedwell et al., 2008
	Management	Unspecified	3	McCartney et al., 2008
Backgrounder or stocker and Feedlot (n = 5)	Feeding	≤ 15	3	Berthiaume, 2006; Şentürklü et al., 2018; Lewis et al., 1990
		Unspecified	2	Gillespie-Lewis et al., 2016
Feedlot (n = 90)	Feeding	≤ 15	8	Price et al., 1978; Xiong et al., 1991; Bartle et al., 1994; Flatt et al., 2003; Sawyer et al., 2004; Swyers et al., 2014; Loerch and Fluharty, 1998; Hinman et al., 1999
		16–50	2	Rivera et al., 2018
		51–350	6	Gallo and Berg, 1995; Vázquez-Añón et al., 2007; Perrett et al., 2008b; Van Donkersgoed et al., 2011; Van Donkersgoed et al., 2014; Sides et al., 2009
		Unspecified	5	Louis et al., 1988; Dubeski et al., 1997; DeRouen and Foster, 2006
	Management	≤ 151	4	Jim et al., 1991; Elam et al., 2008; Step et al., 2008; Cooprider et al., 2011
		51–3502	4	Edwards and Laudert, 1984; Macken et al., 2003; Blaine and Nsahlai, 2011; Kononoff et al., 2015
		Unspecified	1	Kadel et al., 1985
	Multiple	≤15	5	McEwen et al., 2007; Stanton et al., 1989; Sawyer et al., 2003; Lefebvre et al., 2006
		51–350	2	Macken et al., 2003
	Parasite control	≤15	3	Grimson et al., 1987; Campbell et al., 1987; Williams et al., 1991
		16–50	4	Grimson et al., 1987; Alexander and Miller, 1972
		51–350	6	Flack et al., 1967; Schunicht et al., 2000; MacGregor et al., 2001
		≥351	2	Flack et al., 1967
		Pen size varied	1	Guichon et al., 2000
		Unspecified	5	Bauck et al., 1989; Soll et al., 1991; Leland et al., 1980; Wellington and Van Schalkwyk, 1982
	Parenteral antimicrobial	16–50	1	Hannon et al., 2009
		51–350	17	Mechor et al., 1988; Jim et al., 1992; Jim et al., 1999; Schunicht et al., 2002a; Schunicht et al., 2002b; Booker et al., 2007; Van Donkersgoed et al., 2008a; Van Donkersgoed et al., 2008b; Abutarbush et al., 2012; Van Donkersgoed and Merrill, 2012; Van Donkersgoed et al., 2013; Van Donkersgoed and Hendrick, 2013; Van Donkersgoed and Merrill, 2013a; Van Donkersgoed and Merrill, 2013b; Tennant et al., 2014; Van Donkersgoed et al., 2017
		Pen size varied	3	Booker et al., 2006; Schunicht et al., 2007; Perrett et al., 2008a
		Unspecified	2	Stegner et al., 2013; Booker et al., 1992
	Vaccination	51-350	6	Gummow and Mapham, 2000; Schunicht et al., 2003; Bryant et al., 2008; Perrett et al., 2008c; Wildman et al., 2008; Rogers et al., 2015
		Pen size varied	2	MacGregor and Wray, 2004; Wildman et al., 2009
		Unspecified	1	Bechtol et al., 1991

In one trial (Step et al., 2008), 13–16 animals were housed per pen.

One trial (Kononoff et al., 2015) reported 52 head per pen plus or minus 5.69 head.

Characteristics of Trial Reporting

Primary outcome(s) were not stated for 64% (72/113) of the trials; although not as an outcome of interest, an economic/financial outcome was reported for 18% (13/72) of these trials. Primary outcomes were reported for 41 trials; 44% (18/41) included an economic/financial outcome as a primary outcome and 34% (14/41) included an economic/financial outcome as a secondary outcome. An economic/financial outcome was reported, but not stated as a primary or secondary outcome of interest, for nine trials (22%; 9/41). All reported outcomes and how they were reported are presented in Table 4.

Table 4.

The number and percentages of trials by outcomes of interest and how they were reported

Primary outcomes stated (n = 41)
Primary outcome(s)	Secondary outcome(s)	Outcome(s) measured, but not stated as outcome(s) of interest	Outcome(s) used in the economic/financial assessment, but not stated as outcome(s) of interest	N (%)
Economic	-	-	Carcass, Live perf.	5 (12.2)
Health	-	Economic	-	4 (9.8)
Health	Economic	-	-	4 (9.8)
Health	Economic, Live perf.	-	-	4 (9.8)
Economic	-	-	Health, Live perf.	3 (7.3)
Economic	-	-	Live perf.	3 (7.3)
Economic	-	-	Carcass, Health, Live perf.	2 (4.9)
Carcass	Economic	-	-	1 (2.4)
Carcass	Economic, Live perf.	-	-	1 (2.4)
Carcass, Live perf.	Economic	-	-	1 (2.4)
Carcass, Live perf., Health1	-	Economic	-	1 (2.4)
Economic	-	-	Carcass	1 (2.4)
Economic	-	-	Carcass, Health	1 (2.4)
Economic	Health	-	Carcass, Health, Live perf.	1 (2.4)
Economic	Health	-	Live perf.	1 (2.4)
Economic, Health		-	Carcass, Live perf.	1 (2.4)
Health	Carcass, Economic, Live perf., Health2	-	-	1 (2.4)
Health	Live perf.	Economic	-	1 (2.4)
Health	-	Economic, Live perf.	-	1 (2.4)
Health	-	Carcass, Economic, Live perf.	-	1 (2.4)
Health, Live perf.	-	Economic	-	1 (2.4)
Live perf.	Economic, Health	-	-	1 (2.4)
Live perf.	Carcass, Economic, Health	-	-	1 (2.4)

Live perf. = Live Performance

Carcass, live performance, and health were all stated as outcomes of interest, but a primary outcome was not identified.

Two different health outcomes were stated as a primary and secondary outcome of interest.

An economic/financial outcome was stated as a primary outcome of interest and another economic/financial outcome was also reported, but not stated as an outcome of interest.

As part of the data extraction, we used an explicit set of criteria to determine primary and secondary objectives. Whereas this was necessary for systematic extraction, often author language or reporting did not directly fit the terminology or criteria that we had defined. As a result, the 72 trials charted as not stating their primary outcome may be over-represented. However, the ambiguity of reporting the primary objective in 64% of the trials underscores the need for standardized reporting guidelines and the need to adopt the use of such guidelines. These 72 trials belonged to 57 articles, 14 (25%) of which were published after 2010, when REFLECT guidelines were made available. Overall, an economic/financial outcome was reported as an outcome of interest in 91 trials (81%), whereas 22 trials (19%) reported an economic/financial outcome, but did not state that it was an outcome of interest.

The distribution of the reporting of the economic/financial methodology of the trials by whether or not an economic/financial outcome was stated as an outcome of interest is presented in Table 5. In addition to categorizing if the methodology and/or type of economic assessment were described, we also extracted if a statistical test was used to evaluate the economic/financial difference between treatments; a statistical evaluation was used in 44% (50/113) of the trials. The inclusion of a statistical evaluation of the cost/benefit of the treatment, if performed and reported appropriately, is of added value to the stakeholder as it provides estimates and, more importantly, the degree of uncertainty around those estimates.

Table 5.

The number of trials (n = 113) grouped by and across reporting of the economic/financial assessment methodology, type of economic assessment, and if the economic/financial difference between treatments was evaluated with a statistical model by whether the economic/financial outcomes were reported or not as an outcome of interest

	Methodology reported (n = 83)1				Methodology not reported (n = 30)1				Total
Economic/financial outcome stated as an outcome of interest	Economic assessment reported (n = 29)2		Economic assessment not reported (n = 54)2		Economic assessment reported (n = 21)2		Economic assessment not reported (n = 9)2
	Statistically evaluated3	Not statistically evaluated3	Statistically evaluated3	Not statistically evaluated3	Statistically evaluated3	Not statistically evaluated3	Statistically evaluated3	Not statistically evaluated3
Yes	9	14	26	17	0	16	2	7	91
No	1	5	7	4	5	0	0	0	22
Total	10	19	33	21	5	16	2	7

If the methodology used for the economic/financial assessment was reported for the trial.

If the type of economic assessment was reported.

Whether the economic/financial difference between treatment was evaluated using a statistical model.

Of the 22 trials which had economic/financial outcomes reported, but were not stated as an outcome of interest, 17 (77%) reported the methodology for the assessment, whereas 5 (23%) reported only the type of economic assessment used (Table 6). One plausible explanation for this discrepancy is that these studies were not designed to include an economic/financial outcome; as such the outcome and affiliated analysis were included after the study had been completed as an added value after observing differences in health or performance outcomes, or possibly if a reviewer requested an economic/financial value during the publication process. Despite not stating the economic/financial outcome as an outcome of interest, some type of methodology information was reported for all 22 of these trials. Of the nine trials where no methodology was reported, but did state an economic/financial outcome as an outcome of interest (Table 6), only one reported it as the primary outcome. Six of these nine trials came from a single article which only provided some methods for one of the two economic/financial outcomes reported. The four articles that comprised these nine trials were published in years 1967, 1978, 1987, and 2004. Here, the lack of methodology and statement of primary objectives is likely due to poor and unstandardized reporting, especially prevalent in older papers, causing ambiguity and/or discrepancies in the interpretation of the methods and results. Adapting current guidelines, or creating new ones, would be beneficial to researchers and stakeholders as it would support the improvement of economic analyses reporting in livestock research (Totton et al., 2018).

Table 6.

The number of trials for each type of economic assessment by the reported methodology

Economic/financial outcome reported as an outcome of interest?	Economic/ financial assessment methodology reported?	Authors reported the type of economic assessment?1	Partial Budget	Cost/Relative Cost-effectiveness	Enterprise analysis	Cost-benefit	Not enough info.	Break-even analysis	Profit-ability	Other	Total (%)2
Yes	Yes	No	31	0	10	0	2	0	0	0	43 (38.1)
Yes	Yes	Yes	4	3	4	4	0	3	3	23	23 (20.3)
Yes	No	Yes	1	15	0	0	0	0	0	0	16 (14.2)
No	Yes	No	8	0	0	0	2	0	0	14	11 (9.7)
Yes	No	No	8	0	0	0	1	0	0	0	9 (8.0)
No	Yes	Yes	0	2	0	4	0	0	0	0	6 (5.3)
No	No	Yes	0	5	0	0	0	0	0	0	5 (4.4)
No	No	No	0	0	0	0	0	0	0	0	0 (0.0)
		Total (%)2	52 (46.0)	25 (22.1)	14 (12.4)	8 (7.1)	5 (4.4)	3 (2.7)	3 (2.7)	3 (2.7)	113 (100)

If the type of economic assessment was not reported by the authors it was categorized by reviewers with content knowledge.

Percentages are out of the total number of trials 113.

Decision analysis and economic advantage

Gross margin analysis

Answering our first primary objective, to chart the types of economic assessments reported in the literature, there were eight types of economic assessments reported by the authors: cost-effectiveness/relative cost-effectiveness (n = 25), cost–benefit analysis (n = 8) including “cost–profit analysis” and “cost-to-benefit ratio”, enterprise analysis (n = 5), partial budget (n = 5), break-even analysis (n = 3), profitability (n = 3), decision analysis (n = 1), and economic advantage (n = 1). Without clear descriptions of the economic assessments, through definitions and transparent methods, we wanted to avoid over interpretation and chose to only extract the assessments as reported by the articles’ authors. For example, the most commonly reported economic assessment was cost-effectiveness/relative cost-effectiveness. In several of the reported cost-effectiveness/relative cost-effectiveness studies, the authors used monetary units to quantify costs and outcomes (i.e., a cost–benefit analysis), where a cost-effectiveness analysis would use an appropriate natural nonmonetary effect (Babo Martins and Rushton, 2014). It was beyond the scope of this review to confirm whether the methods employed and type of assessment reported conformed to conventional use of these terms in the economic literature; however, after a brief perusal of this information, we found discrepancies in the use of the terminology employed to characterize economic assessments.

After evaluating the methods from trials without a specific economic assessment reported, we identified three types that were used: partial budget (n = 47), enterprise analysis (n = 10), and gross margin analysis (n = 1). There was not enough information provided in the article to determine the type of economic assessment used in five trials. The type of economic assessment by the reporting of the economic/financial outcome and methodology is presented in Table 6. Some studies lacked descriptions or had incomplete descriptions of economic data, including not reporting economic values and not identifying the data source, which created difficulties for identifying the type of assessment used. In addition, the classification of the type of economic assessment for these studies may be biased as it was conducted, though by a context expert (i.e., agricultural economist), via a sole reviewer.

The types of economic assessments used by intervention, measured animal outcome, and outcome utilization are presented in Table 7. All measured outcomes were used to estimate the economic/financial consequences of the intervention in 42% (47/113) of the trials and two types of economic assessment were primarily used, partial budget (27/47) and enterprise budget analysis (9/47); more than one type of animal outcome was measured in 72% (34/47) of these trials. Only some of the measured animal outcomes were analyzed in 31% (35/113) of the trials and a partial budget analysis was the most common economic assessment (57%; 20/35). In only 14% (5/35) of these trials the measured animal outcome(s) used in the economic/financial analysis was reported as primary outcome(s). The outcomes of primary interest were not stated in the other 86% (30/35) of these trials. As with any study, identifying the primary outcome(s) and utilizing an appropriate sample size are crucial for accurately and precisely estimating the magnitude of the effects between interventions and outcomes (Sargeant et al., 2010), and economic analyses are no different. Stating if the economic/financial outcome is a primary or secondary objective of the study would allow the researchers to ensure that the study would be appropriately sized for accurate estimation and proper inference.

Table 7.

The number of trials by economic assessment, intervention, measured animal outcomes, and outcome utilization and reporting

Economic assessment (Citations)	Intervention	All animal outcomes measured	Animal outcomes utilized2	Primary?3	N
Break-even analysis1 (n = 3)
(Peterson et al., 1989; Lewis et al., 1990)	Feeding (n = 2)	Carcass, Live perf.	All	-	1
		Live perf.	All	-	1
	Management (n = 1)	Live perf.	All	-	1
Cost–benefit analysis1 (n = 8)
(Kadel et al., 1985; Bechtol et al., 1991; Soll et al., 1991; Xiong et al., 1991; Blaine and Nsahlai, 2011; Van Donkersgoed et al., 2013; Van Donkersgoed and Hendrick, 2013)	Feeding (n = 1)	Carcass, Live perf.	Some	NR	1
	Management (n = 2)	Carcass, Live perf.	Some	NR	1
		Health, Live perf.	Some	NR	1
	Parasite control (n = 1)	Carcass	All	-	1
	Parenteral antimicrobial (n = 2)	Health	None	-	2
	Vaccination (n = 2)	Health	All	-	2
Cost-effectiveness/Relative cost-effectiveness1 (n = 25)
(Bauck et al., 1989; Jim et al., 1999; Guichon et al., 2000; Schunicht et al., 2000; Schunicht et al., 2002a; Schunicht et al., 2002b; Schunicht et al., 2003; Booker et al., 2006; Booker et al., 2007; Schunicht et al., 2007;Perrett et al., 2008a; Perrett et al., 2008b; Perrett et al., 2008c; Van Donkersgoed et al., 2008a; Van Donkersgoed et al., 2008b; Wildman et al., 2008; Hannon et al., 2009; Wildman et al.,2009; Abutarbush et al., 2012; Van Donkersgoed and Merrill, 2012; Van Donkersgoed and Merrill, 2013a; Van Donkersgoed and Merrill, 2013b; Tennant et al., 2014; Van Donkersgoed et al., 2017)	Feeding (n = 1)	Carcass, Health, Live perf.	Significant only	NR	1
	Parasite control (n = 3)	Carcass, Health, Live perf.	Significant only	NR	1
		Health, Live perf.	Some	Yes	1
		Live perf.	All	-	1
	Parenteral antimicrobial (n = 17)	Carcass, Health, Live perf.	All	-	1
			Significant only	No	1
				NR	2
				Yes	3
		Health	Significant only	NR	1
				Yes	1
		Health, Live perf.	Significant only	No	1
				NR	1
				Yes	6
	Vaccination (n = 4)	Carcass, Health, Live perf.	Significant only	NR	4
Decision analysis1(n = 1)
(Booker et al., 1992)	Parenteral antimicrobial (n = 1)	Health	All	-	1
Economic advantage1 (n = 1)
(Edwards and Laudert, 1984)	Management (n = 1)	Carcass, Health, Live perf.	Some	NR	1
Enterprise analysis (n = 14)
(Hinman et al., 1999; Sawyer et al., 2004; Anderson et al., 2005; DeRouen and Foster, 2006; Bedwell et al., 2008; Kirkpatrick et al., 2008; McCartney et al., 2008; Mulliniks et al., 2012; Parish et al., 2013; Şentürklü et al., 2018)	Feeding (n = 9)	Carcass, Health	Some	NR	3
		Carcass, Live perf.	All	-	2
		Carcass, Live perf.	Some	NR	1
		Carcass, Health, Live perf.	All	-	3
	Management (n = 3)	Carcass, Live perf.	All	-	3
	Multiple (n = 1)	Carcass, Health, Live perf.	All	-	1
	Vaccination (n = 1)	Carcass, Health, Live perf.	Some	NR	1
Gross Margin analysis (n = 1)
(Lefebvre et al., 2006)	Multiple (n = 1)	Carcass, Live perf.	Some	NR	1
Not Enough Information (n = 5)
(Williams et al., 1991; Loerch and Fluharty, 1998; Flatt et al., 2003; MacGregor and Wray, 2004; Mir et al., 2008)	Feeding (n = 3)	Carcass	Some	Yes	1
		Carcass, Live perf.	Some	NR	1
		Carcass, Health, Live perf.	Some	NR	1
	Parasite Control (n = 1)	Health, Live Perf.	Not reported	-	1
	Vaccination (n = 1)	Carcass, Health, Liveperf.	Some	NR	1
Partial Budget (n = 52)
(Flack et al., 1967; Alexander and Miller, 1972; Price et al., 1978; Leland et al., 1980; Wellington and Van Schalkwyk, 1982; Campbell et al., 1987; Grimson et al., 1987; Karren et al., 1987; Louis et al., 1988; Mechor et al., 1988; Stanton et al., 1989; Jim et al., 1992; Bartle et al., 1994; Gallo and Berg, 1995; Dubeski et al., 1997; Jim et al., 1999; Gummow and Mapham 2000; MacGregor et al., 2001; Macken et al., 2003;Sawyer et al., 2003;Berthiaume, 2006;Stalker et al., 2006;McEwen et al., 2007; Shike et al., 2007; Vázquez-Añón et al., 2007; Bryant et al., 2008; Elam et al., 2008; Step et al., 2008;Larson et al., 2009;Sides et al., 2009; Cooprider et al., 2011; Van Donkersgoed et al., 2011; Swyers et al., 2014; Van Donkersgoed et al., 2014; Kononoff et al., 2015; Rogers et al., 2015; Forte et al., 2018; Rivera et al., 2018)	Feeding (n = 16)	Live perf.	None	-	1
		Carcass, Live perf.	All	-	3
			Some	NR	2
			Significant only	NR	1
			Not reported	-	1
		Health, Live perf.	All	-	1
			Some	-	2
		Carcass, Health, Live perf.	Some	NR	4
			Significant only	NR	1
	Management (n = 9)	Carcass, Live perf.	All	-	2
			Some	NR	2
		Carcass, Health, Live perf.	All	-	1
			Some	NR	2
		Health, Live perf.	All	-	2
	Multiple (n = 6)	Carcass, Live perf.	All	-	3
			Some	NR	3
	Parasite Control(n = 16)	Carcass, Health	Some	NR	1
		Carcass, Health, Live perf.	Some	Yes	1
			Significant only	NR	1
		Health, Live perf.	All	-	6
		Live perf.	All	-	7
	Parenteral antimicrobial (n = 2)	Health	Some	Yes	2
	Vaccination (n = 3)	Carcass, Health	All	-	1
		Carcass, Health, Live perf.	All	-	1
			Some	NR	1
Profitability1 (n = 3)
(Stegner et al., 2013; Gillespie-Lewis et al., 2016)	Feeding (n = 2)	Carcass, Live perf.	All	-	2
	Parenteral antimicrobial (n = 1)	Carcass, Health, Live perf.	Not reported	-	1

Live perf. = Live performance; NR = Not reported.

Includes only author reported types of economic assessment.

Which of the measured animal outcomes were used to derive the cost/benefit of the intervention.

If the animal outcomes used to determine the cost/benefit of the intervention were included in the authors’ stated primary objectives. Only “significant only” or “some” animal outcome utilized were extracted. If the authors did not report their primary objective(s), it was extracted as “not reported.”

In three trials, only the cost of the intervention, but no health, performance, or carcass outcomes, was reported; a cost–benefit analysis was performed in two of these trials and the investigators a priori planned to utilize only significant outcomes in the analysis (although none were significant) (Table 7). The outcomes used to estimate the cost and benefits of the intervention were not reported in three of the trials (Table 7). Only animal outcomes that were significantly different between treatments were analyzed in 22% (25/113) of the trials and a cost-effectiveness or relative cost-effectiveness analysis was performed for 88% (22/25) of these trials (Table 7). The significantly different animal outcomes used in the economic/financial analysis were stated as primary objectives in 40% (10/25) of these trials, in 8% (2/25) the animal outcomes used were not the primary outcomes of interest, and in 52% (13/25) of the trials the outcomes of primary interest were not stated (Table 7). Utilizing a subset of the results to calculate the economic benefit and/or cost of the intervention may lead to an over or under-estimation of the value in part due to type I and type II errors incurred when analyzing multiple outcomes before performing the economic assessment.

All or some of the values used to estimate the economic/financial outcome were reported in 86% (97/113) of the trials. The sources for all or some of the values used to estimate the economic/financial outcome were reported in 57% (64/113) of the trials. Only private sources were cited in 55% (35/64) of these trials, both private and public sources were cited in 28% (18/64), only public sources were cited in 11% (7/64), public sources and commercial services were cited in 5% (3/64), and only commercial services were cited in a single trial (2%). The date of the study or the date the values were sourced was reported in 39% (44/113) of the trials (Table 8). While it is understandable that detailed financial information from commercial feedlots may not be publicly available, baseline data on values utilized in calculations are necessary for researchers to assess validity and reproducibility (Sargeant et al., 2010).

Table 8.

The number of trials, percentages, and types of sources grouped by the reporting of the economic data used in the economic/financial assessment by different types of methodology reporting

Economic/financial outcome stated as an outcome of interest?	Economic/financial assessment methodology reported?	Type of Economic assessment reported?	Value, Source and Date1	Value and Source1	Source and Date1	Value and Date	Value	Source1	No info.	Total (%)2
						N
Yes	Yes	Yes	15(A)	1(PP)	1 (PR)	0	4	0	2	23 (20.3)
Yes	Yes	No	15(A)	17(PP)	3 (PR)	0	6	1 (PR)	1	43 (38.0)
Yes	No	Yes	2(PR)	0	0	0	14	0	0	16 (14.2)
Yes	No	No	0	1 (PR)	1 (PR)	0	1	0	6	9 (8.0)
No	Yes	Yes	2(PP)	0	1(PR)	1	2	0	0	6 (5.3)
No	Yes	No	3(PP)	1 (PR)	0	0	7	0	0	11 (9.7)
No	No	Yes	0	0	0	0	5	0	0	5 (4.4)
No	No	No	0	0	0	0	0	0	0	0 (0.0)
		Total (%)2	37 (32.7)	20 (17.7)	6 (5.3)	1 (0.9)	39 (34.5)	1 (0.9)	9 (7.1)	113 (100)

Acronyms in parentheses indicate the types of sources utilized: A, all: private, public and/or commercial services; CS, commercial service; PP, private and/or public; PR, private; PU, public.

Percentages are out of the total number of trials 113.

An economic/financial outcome stated as an outcome of interest, the methodology used and/or the type of method, and the values, sources, and dates of some or all of the information used in the economic/financial assessment were reported in 28% of the trials (32/113; Table 8). When considering trials where the methodology and supporting information were reported, but the economic/financial outcome was not stated as an outcome of interest (5/113; Table 8), only about a third of the trials met the minimum requirements needed to reproduce the trial results as they pertain to the economic/financial assessment. Not reporting the values used in the economic assessments and their sources limits the inference and external validity of the study and creates a barrier to reproducibility. In other words, lack of reporting limits the value of the publication for end-users (Sargeant et al., 2010). In addition, the increase in the sheer number of studies being published necessitates the use of knowledge synthesis tools, such as scoping reviews, systematic reviews, and meta-analyses, for proper contextualization of the evidence from studies and without thorough reporting, the ability to use these tools is hampered (Page et al., 2021). Given the importance of economic outcomes on the management of livestock production systems, further work is warranted on creating guidelines or standards of practice to improve reporting of economic assessments from livestock field trials.

Click here for additional data file.