Usefulness of applying research reporting guidelines as Writing Aid software: a crossover randomised controlled trial.

OBJECTIVES: To assess the intention of using a Writing Aid software, which integrates four research reporting guidelines (Consolidated Standards of Reporting Trials, Preferred Reporting Items for Systematic Reviews and Meta-Analyses, Strengthening the Reporting of Observational Studies in Epidemiology and STrengthening the Reporting of Observational Studies in Epidemiology-nutritional epidemiology) and their Elaboration & Explanation (E&E) documents during the write-up of research in Microsoft Word compared with current practices.
DESIGN: Two-arms crossover randomised controlled trial with no blinding and no washout period.
SETTING: Face-to-face or online sessions. PARTICIPANTS: 54 (28 in arm 1 and 26 in arm 2) doctoral and postdoctoral researchers.
INTERVENTIONS: Reporting guidelines and their E&E document were randomly administered as Writing Aid or as Word documents in a single 30 min to 1 hour session, with a short break before crossing over to the other study intervention. PRIMARY AND SECONDARY OUTCOMES: Using the Technology Acceptance Model, we assessed the primary outcome: the difference in the mean of intention of use; and secondary outcomes: the difference in mean perceived ease of use and perceived usefulness. The three outcomes were measured using questions with a 7-point Likert-scale. Secondary analysis using structural equation modelling (SEM) was applied to explore the relationships between the outcomes.
RESULTS: No significant difference in reported intention of use (mean difference and 95% CI 0.25 (-0.05 to 0.55), p=0.10), and perceived usefulness (mean difference and 95% CI 0.19 (-0.04 to 0.41), p=0.10). The Writing Aid performed significantly better than the word document on researchers' perceived ease of use (mean difference and 95% CI 0.59 (0.29 to 0.89), p<0.001). In the SEM analysis, participants' intention of using the tools was indirectly affected by perceived ease of use (beta 0.53 p=0.002).
CONCLUSIONS: Despite no significant difference in the intention of use between the tools, administering reporting guidelines as Writing Aid is perceived as easier to use, offering a possibility to further explore its applicability to enhance reporting adherence. © Author(s) (or their employer(s)) 2019. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

RCT Entities:   Population Interventions Outcomes

We developed and tested a novel offline Writing Aid, for authors to use reporting guidelines while writing research findings, using a validated measure for the intention of use.

This is the first study to test the application of reporting guidelines in a real-life setting with a diverse group of participants, including researchers from low/middle-income countries, reporting results from a variety of study designs.

The Writing Aid software was a prototype and improvements are required to resolve technical errors.

The subjective nature of outcomes, short exposure to the intervention and the no washout period between the applications of both tools are limitations.

The study did not assess completeness of reporting and further assessment is necessary in this regard.

Introduction

Incomplete reporting of study results in biomedical research is considered unethical and a waste of (often public) resources.1 A way to increase the usefulness of research is to ensure that all essential information is included in a research manuscript.

Over the last decades, reporting guidelines have been developed and used to increase the completeness and transparency of research findings. A reporting guideline is commonly organised as a checklist of essential items that should be addressed when reporting research manuscripts, in combination with a flow diagram that specifies the items to be reported during the write up of the study.2 Reporting guideline’s Elaboration & Explanation (E&E) documents provide additional explanation and examples of the recommendations.2

The publication of the Consolidated Standards of Reporting Trials ‘CONSORT’ in 1996,3 was followed by a steady increase in reporting guidelines development for different types of study designs including the Preferred Reporting Items for Systematic Reviews and Meta-Analyses ‘PRISMA’ Statement4 and the Strengthening the Reporting of Observational Studies in Epidemiology ‘STROBE’ Statement.5 Extensions of reporting guidelines have also been developed for specific fields, such as the STrengthening the Reporting of Observational Studies in Epidemiology-nutritional epidemiology (STROBE-nut), which aims to improve the completeness of reporting for nutrition research.6 Currently, there are >400 reporting guidelines indexed by the EQUATOR Network, an international organisation that promotes the use of reporting guidelines.7

However, present use of reporting guidelines requires consideration. Reporting guidelines are typically applied at the final stages of the writing process to address journal requirements.8 As a result, reporting guidelines might be considered as an administrative burden rather than a tool to improve research quality. Moreover, research on the usefulness of reporting guidelines from the points of view of the authors is scarce. Previous studies have focused on the completeness of reporting as an outcome measure, which is tailored differently to each reporting guideline,9 10 rather than their usefulness. Perceived intention of use can give an indication of researchers’ willingness to adhere to reporting guidelines.

Long-term adherence to reporting guidelines will depend on how well they are integrated into day-to-day practices and workflows of researchers during the writing process.11 In a survey conducted in 2012, among systematic review authors to test a PRISMA extension, authors recommended the integration of the reporting guideline elements into a systematic review software.12 There have been initiatives attempting to develop new tools and test their impact on reporting guideline adherence. For example, the online COBWEB tool9 13 guides authors on how to apply the CONSORT reporting guideline to their manuscript and Penelope, an automated online tool, generates automatic checks of manuscripts written in MS Word. Penelope is currently being integrated and tested in an online journal submission application.14–16 A recent study also developed a writing tool, and a template with the minimum amount of information to report regarding data handling of biomarkers in metabolomics.17 Nevertheless, none of these efforts focuses on the uptake of several reporting guidelines during the writing process, using common offline writing platforms such as Microsoft Word.

In recognition of these issues, we developed a Writing Aid tool that integrates the reporting guidelines and their E&E documents, in the form of an Add-in for Microsoft (MS) Word (V.1.0, Automaticals Consulting),18 and assessed participants’ intention to use it during the writing process, using the Technology Acceptance Model (TAM). The TAM model has been validated and applied previously to test software in similar settings. It has been used in office environment operations (eg, text editor, voicemail), software application development (eg, software maintenance tool) and core business process software (eg, production control tools).19 The overall objective was to investigate researchers’ intention of using the reporting guidelines as a Writing Aid in Word versus the traditional approach of a Word document and the E&E document. Secondary objectives included perceived usefulness and perceived ease of use. We also assessed how perceived usefulness and perceived ease of use were associated with the intention of use. The questionnaires contained questions with a 7-point Likert-scale response ranging from extremely unlikely (‘1’) to extremely likely (‘7’) (online supplementary appendix 1). The intention of use outcome was constructed from two questions:

Q1—Assuming I have access to the reporting guidelines documents (as a MS Word table and elaboration and explanation document/Writing Aid), I intend to use it.

​Q2—Given access to the reporting guidelines documents (as a MS Word table and elaboration and explanation document/Writing Aid), I predict that I would use it.

The perceived usefulness outcome was constructed from four questions. Participants were asked to rate the reporting guideline usefulness (using the Writing Aid and Word Document) based on the tool’s ability to improve completeness of reporting, increase productivity, enhance effectiveness and usefulness. The perceived ease of use outcome was also constructed from four questions. Participants were asked to rate the ease of using the reporting guidelines (using the Writing Aid and Word Document) based on how flexible, easy to use, easy to provide guidance, clear and understandable it was to interact with.

The study was conceived by authors of STROBE-nut, as an approach to improve its uptake. However, during the study set up, and the software development, it became clear that the intention of use and the software developed are relevant for other reporting guidelines. As a proof-of-concept study, CONSORT, PRISMA and STROBE were included to test the wider application within other research designs and fields.

Methodology

Study design and participants

We performed a randomised controlled crossover trial comparing two ways (tools) of administering research reporting guidelines and their E&E documents; that is, using the traditional Word checklists and documentation (Control: Word Document) versus using the Writing Aid software V.1.0 (intervention: Writing Aid).

Due to the crossover nature of the study design, each participant tested both tools in one of the two alternative sequences representing the two study arms. Arm 1 participants received the Writing Aid first followed by the Word Document and arm 2 participants received the Word Document first followed by Writing Aid. Participants were assigned to one of the two study arms randomly. For this purpose, at the start of the study, the lead investigator (DH) generated a randomisation list using Microsoft Excel, where numbers from 1 to 100 were randomised into either arm 1 or arm 2. Then, each randomisation number with its corresponding arm were written on a piece of paper, folded and put into a box to be picked by participants on the study day. There was no washout period between the testing of the two tools. Neither the study participants nor the researchers were blinded to the sequence of the intervention allocation, or assessment of outcomes. No formal sample size calculation was conducted and we aimed to collect as many responses as possible. This study was reported using the CONSORT recommendations20 (see online supplementary appendix 2).

Purposive sampling and snowballing was used to recruit participants from May until the end of October 2018. Eligible subjects were doctoral and postdoctoral researchers who were writing or had recently published a paper in any biomedical research field in the previous 6 months. Personalised email invitations were disseminated to potentially eligible students at Ghent University, the University of Split, the Methods in Research on Research ‘MiRoR’ network, and at conferences (Federation of European Nutrition Societies, Belgrade 2018, Tropentag Gent 2018, The Cochrane Colloquium 2018). Twitter and posters were also used to circulate the invitation to a wider audience.21

Tools

Writing Aid

The Writing Aid software was developed as a Microsoft Word Add-in in Visual Basic and it works offline on all versions of Microsoft Office, operating on a Windows system (Writing Aid software V.1.0, Automaticals Consulting).18 For each checklist, the tool generates a specific checklist table, dropdown menu options containing the reporting requirements, and an information box that contains the text of the E&E manuscript for each checklist. The tool has the following functionalities:

Users can select a checklist applicable to their manuscript. Once selected, a reporting table is automatically added at the end of the manuscript.

Authors can annotate manuscript text (right mouse click) and tag it to the corresponding item of the checklist.

The annotation is visually displayed in the margins (similar to the Comments function in Word document) with the tagged text automatically copied into the reporting table at the end of the paper. When annotated text is edited, it is also updated in the table.

After completing the annotation process, users have the option to fill in the remaining blank items in the reporting table and provide additional explanations why certain items are not reported.

The flowcharts of PRIMSA and CONSORT were not included. The decision was made as the study mainly focused on the writing process. Although flowcharts provide crucial information for the manuscript, they are not typically part of the narrative sections. Moreover, their inclusion requires further sophistication of software programming, which time and resources did not allow. The user manual can be found on GitHub18 and in online supplementary appendix 3.

Traditional tool-Word document

For the control (Word Document) tool, we used the relevant checklists of reporting guidelines and their E&E document which were downloaded from the relevant websites.22–25

Study procedures

The study was administered in the computer labs of Ghent University under the supervision of the lead investigator (DH). In the protocol, it was planned to conduct all sessions face-to-face. However, to recruit as many researchers as possible we used video calls through Skype for those residing outside Ghent. On the testing day, participants drew a randomisation code. When the study was done remotely using Skype video call, the lead investigator (DH) picked the piece of paper containing the code.

Participants could select whichever paper, with a relevant study design (systematic reviews, observational studies or randomised controlled trials), to test the tools.

Prior to the application of the Writing Aid, the lead investigator (DH) ensured the Writing Aid was correctly installed and functional. There was minimal social interaction with participants during the study to minimise social desirability bias. Apart from resolving technical errors, no additional assistance related to the study or use of checklists was provided. When technical errors could not be resolved, another computer was provided or participants were asked to use a different device if they were participating remotely.

After allocation, participants completed a baseline questionnaire and read a half page explanatory document (online supplementary appendix 1). The document included a list of points that summarised the concept of reporting guidelines. There were no clarifications regarding the content of reporting guidelines. Participants worked at their own pace and had a maximum of 1 hour to test each tool.

In arm 1, participants applied the Writing Aid to their document first. If they wanted to access the E&E document, they could use the information box. A user manual and a 3 min video on the functionalities of the tool were provided.18 In arm 2, participants manually applied the reporting guidelines as a Word Document by inserting the page number where the relevant information could be found in their manuscript. They were also given the E&E document.

On completion of testing the first tool, participants were asked to complete the first evaluation questionnaire (online supplementary appendix 1 with questionnaires). A break of a few minutes was given, and then participants began the test of the other tool. The second evaluation questionnaire was administered after the last test.

Survey instruments

Study outcomes and measurement

The primary study outcome was subjects’ intention of using the reporting guideline as Writing Aid and Writing Document. Secondary outcome measures included subjects’ perceived ease of use and perceived usefulness of the two tools.

Self-administered structured questionnaires applied via Qualtrics (Qualtrics XM, Provo, Utah, USAT) were used to collect data on study outcome measures and other relevant variables. All questionnaires were piloted by the primary investigators (DH, MKS and CL) for clarity. The study outcomes, that is, intention of use, perceived ease of use and perceived usefulness, were measured using validated instruments adapted from the TAM.26 27

In addition to the three outcomes based on the TAM instrument, the respondents’ preferred tool for later use and the occurrence of technical errors encountered were assessed.

Baseline and relevant characteristics of participants were gathered, including their research experience, role in the study used, study design of the manuscript tested and previous experience with reporting guidelines. Previous experience included previous use, frequency of use and motivation of use. We also assessed participant’s prior knowledge regarding reporting guidelines using a validated tool to assess knowledge regarding checklists.28 Subjective knowledge, considering the utilisation and content of the reporting guidelines, was measured with two questions on a 5-point Likert scales ranging from very unknowledgeable (1) to very knowledgeable (5). Objective knowledge was measured using six true or false statements. Three true statements were included (1) it is acceptable to report that some items on the checklist are not applicable to my study; (2) reporting on items that are not carried out will add more clarity to my paper and will not lead to rejection; (3) the checklists aim to make reporting more clear, complete and transparent. The three false statements were (1) the checklist should be used to evaluate the quality of papers; (2) the reporting checklists must be completely filled out, or my paper will be rejected; (3) the checklist aims to improve communication between coauthors.

Statistical methods

Data cleaning and analysis were conducted using Stata V.14.1 (StataCorp). All analyses were two-sided and statistical significance was considered at alpha <0.05. Data were checked for consistency, missing values, outliers and normality prior to analysis. Descriptive statistics were reported using percentage and mean with SD.

For the main analysis of the intervention effects on the outcome variables intention of use, perceived ease of use and perceived usefulness, we analysed the data according to the two period crossover trial design. Prior to testing the treatment effect, we confirmed the absence of any potential sequence or period effect using independent-sample t-test and paired-sample t-test, respectively. The intervention effect was estimated by looking at the average of the treatment difference for each period using paired-sample t-test.29 We used the t-test to test the difference in mean intention of use, perceived usefulness and ease of use after confirming normality of data. The intervention effect-size for the difference between Writing Aid and Word Document was reported using Cohen’s d (mean difference/SD) with values ≤0.2, 0.2> and <0.8, and ≥0.8 considered as small, medium and large intervention effects, respectively.30

To provide an explanation for participants’ intention of using the tools (Writing Aid vs Word Document), which is related to perceived ease of use and/or perceived usefulness, we also conducted structural equation modelling (SEM) guided by the TAM (figure 1). Based on TAM, we hypothesised that the use of the Writing Aid for reporting guidelines would result in increased subjects’ intention of use compared with the use of the Word Document, which could be facilitated through: (1) immediate pathway between subjects’ better perceived ease of use for the Writing Aid compared with the Word Document leading to a better intention of use and (2) chain pathway in which subjects’ better perceived ease of use could lead to a better perceived usefulness of the Writing Aid compared with the Word Document and finally result in a better intention of use. We assumed that perceived usefulness would not be affected by the intervention used, as the same checklist content was applied in both arms. SEM with maximum likelihood estimation was fitted to model the hypothesised relationships described above. In the measurement models, factor analysis was employed to estimate the latent variables intention of use, perceived usefulness and perceived ease of use from their construct observed variables. We estimated both unstandardised and standardised estimates of the direct and indirect effect of the treatment (Writing Aid vs Word Document) on intention of use through the hypothesised pathways. We evaluated the reliability of the measurement scales and the relative importance of each construct variable in a scale using Cronbach’s alpha coefficient (alpha >0.7 was considered acceptable),31 item-total correlation coefficients and factor loadings. Model goodness-of-fit was checked using fit statistics including the Comparative Fit Index >0.95, Tucker-Lewis Index>0.95, Standardised Root Mean Squared Residual <0.08, Root Mean Square Error of Approximation <0.06.32

Figure 1

Technology Acceptance Model hypothesised pathways of intervention effect on intention of use: direct, indirect.

Ethics

Informed consent was electronically collected and the study protocol was registered prior to the study (20 April 2018).33

Patient and public involvement

We did not involve patients or the public in our study.

Results

Participants

We recruited 54 participants between May and October 2018, of which 28 and 26 were randomly allocated in arms 1 and 2, respectively; all participants completed the trial (figure 2). It was not possible to assess response rate, as recruitment methods used a snowballing approach. However, in this study only those who willingly wanted to participate n=54 completed the study.

Figure 2

Participants flowchart. Period 0, period 1 and period 2 represent the baseline, first test and second test data collection, respectively.

As shown in table 1, 80% (n=42/54) of the sample was PhD students and nearly all (n=50/54, 93%) were the first author of the manuscript. Over half (n=33/54, 61%) reported findings of an observational study, (n=11/54, 20%) a randomised controlled trial and (n=10/54, 19%) a systematic review. Half of the sample (n=27) had never used any reporting guideline before and almost half of the sample (n=25) considered themselves unknowledgeable regarding reporting guidelines’ content or their utilisation (n=26). Only 17% (n=9/52) correctly answered that reporting guidelines should not be used as an evaluation tool for the quality of the paper. Almost all participants (94% and 91 %) correctly answered the two statements regarding the aim of the reporting guidelines.

Table 1

Sample characteristics

Sample characteristics	N (%)
Research experience
PhD student	43 (80)
Post-doctoral student	11 (20)
Affiliation regarding the current paper
First author	50 (93)
Coauthor	4 (7)
Study design
Systematic review	10 (19)
Randomised controlled study	11 (20)
Observational study (cross-sectional, cohort, case-control)	33 (61)
Previous reporting guidelines use*
No, it will be my first time to use reporting guidelines	27 (50)
Yes, to write or co-write a paper	13 (22)
Yes, to write this paper	11 (17)
Yes, to review a paper	2 (2)
Frequency of reporting guidelines use
Never	19 (35)
Rarely	12 (22)
Sometimes	9 (17)
Usually	12 (22)
Every time	2 (4)
Motivation of guideline use*†
Self-motivation or motivation from colleagues or coauthors	12 (22)
Journal suggestions to use checklists within the writing process	1 (2)
Journal requirements to fill the checklist at the end	5 (9)
Subjective knowledge
How do you rank your knowledge with respect to the content of the reporting guideline?‡
Very knowledgeable	3 (6)
Somewhat knowledgeable	17 (31)
Neither knowledgeable nor unknowledgeable	8 (15)
Somewhat unknowledgeable	10 (19)
Very unknowledgeable	15 (28)
How do you rank your knowledge with respect to the utilisation of the reporting guideline?
Very knowledgeable	2 (4)
Somewhat knowledgeable	17 (31)
Neither knowledgeable nor unknowledgeable	9 (17)
Somewhat knowledgeable	11 (20)
Very unknowledgeable	15 (28)
Objective Knowledge
Answer the following statement with true or false (frequency of the correct answer)
The checklist should be used to evaluate the quality of papers* ( FALSE)	9 (17)
The reporting checklists must be completely filled, or my paper will be rejected† ( FALSE)	37 (69)
It is acceptable to report that some items on the checklist are not applicable to my study* ( TRUE)	49 (91)
Reporting on items that are not carried out will add more clarity to my paper and will not lead to rejection* ( TRUE)	36 (69)
The checklists aim to make reporting more clear, complete and transparent*(TRUE)	51 (94)
The checklist aim to improve communication between coauthor* ( FALSE)	34 (63)

*Indicate a multiple-response question.

†n = 27

‡n=53

Outcomes

We did not find a significant sequence or period effect (p>0.05) in the crossover design. Table 2 shows that there was no significant difference in the performance between the Writing Aid and the Word Document for both the primary outcome; intention of use (mean difference and 95% CI 0.25 (-0.05 to 0.55), p=0.10), and perceived usefulness (mean difference and 95% CI 0.19 (-0.04 to 0.41), p=0.10). A significant effect was found when comparing the perceived ease of use of the Writing Aid compared with the Word Document (mean difference and 95% CI 0.59 (0.29 to 0.89), p<0.001).

Table 2

Effect of the intervention on primary and secondary outcomes, mean (SD), comparing Writing Aid and MS Word tools in a crossover design (n=54)

Outcomes (factor score)	MS WordMean (SD)	Writing AidMean (SD)	Mean difference and 95% CI*	P value of mean difference	Effect-size 95% CI*
Intention of use	5.51 (1.24)	5.84 (1.24)	0.25 (-0.05 to 0.55)	p=0.10	0.23 (-0.05 to 0.5)
Perceived usefulness	5.38 (1.14)	5.63 (1.06)	0.19 (-0.04 to 0.41)	p=0.10	0.23 (-0.04 to 0.5)
Perceived ease of use	5.25 (1.30)	5.98 (0.93)	0.59 (0.29 to 0.89)	p<0.001	0.54 (0.25 to 0.83)

*Cohen’s d values used to estimate the effect size for the difference between the interventions (ie, Writing Aid minus MS Word documentation scores) in terms of SD scores; Cohen’s d values (x≤0.2, 0.2

In the present sample, the Cronbach’s alphas were 0.87, 0.89 and 0.97 for perceived usefulness, perceived ease of use and intention of use, respectively (online supplementary appendix 4). SEM indicated an acceptable goodness of fit, as the Comparative Fit Index and Tucker-Lewis Index were both above 0.95. The standardised root mean squared residual was below the cut-off of 0.08, but the root mean square error of approximation was not below its cut point of 0.06. Based on Schreiber et al, all reported measures indicate that there is an acceptable goodness of fit between our data and the model.32

As shown in table 3, the total effect of the Writing Aid on intention of use was significantly mediated through higher perceived ease of use vs the Word Document (beta coefficient 0.5, p=0.02). The direct component was negative 0.03; by contrast the indirect effect was 0.53, indicating that the effect of the Writing Aid on the intention of use was fully arbitrated by perceived ease of use. The total effect of the perceived ease of use (Writing Aid vs Word Document) on intention of use was 0.92. An estimated 25% (0.23/0.92) of the effect of perceived ease of use on intention of use is direct, while 75% of the effect was indirect and was mediated through perceived usefulness. A significant indirect effect of the tools (Writing Aid vs Word Document) on perceived usefulness mediated through perceived ease of use was observed.

Table 3

Structural equation modelling: parameter estimates for the hypothesised pathways: direct, indirect and total effects, beta coefficient and p values

Hypothesised pathway	Standardised estimate
	Direct effectbeta coefficient (SD)	P value	Indirect effectbeta coefficient (SD)	P value	Total effectbeta coefficient (SD)	P value
StructuralPU <- PEU	0.56 (0.11)	<0.001*			0.56 (0.11)	<0.001*
PU <- Intervention effect			0.33 (0.11)	0.003*	0.33 (0.11)	0.003*
PEU <- Intervention effect	0.60 (0.18)	0.001*			0.60 (18)	0.001*
IU <- PU	1.23 (0.21)	<0.001*			1.23 (0.21)	<0.001*
IU <- PEU	0.23 (0.14)	0.11	0.69 (0.14)	<0.001*	0.92 (0.15)	<0.001*
IU <- Intervention effect	−0.03 (0.16)	0.87	0.53 (0.17)	0.002*	0.50 (0.21)	0.02*

Goodnessof fit results R2: R-squared = 0.145; standardised root mean squaredresidual = 0.048, root mean square error of approximation = 0.074, CFI = 0.975, TLI = 0.965.

CFI, Comparative Fit Index; IU, intention of use; PEU, perceived ease of use; PU, perceived usefulness; TLI, Tucker-Lewis Index.

Other measures

More than two-third of the sample (n=42, 77%) selected the Writing Aid as the preferred method of use for later use. Almost one-third of the study sample (n=17, 32%) encountered a technical issue when installing the Writing Aid.

Discussion

This study attempted to test the intention to use of a novel Writing Aid software vs the traditional Word Document version of several widely used reporting guidelines. This paper extends prior knowledge by using an intervention to test the uptake of reporting guidelines in a real-life writing process, using all sections of a paper.

In the present study, participants indicated no significant difference in intention of use, and perceived usefulness between the two tools. This can be explained by the fact that the two applied interventions contained the same recommendations for reporting. However, participants perceived the Writing Aid to be easier to use than the Word Document with a significant effect. This can be attributed to the difference in application characteristics (integrated software vs a MS Word document). The results further show that the perceived ease of use of the guidelines as a Writing Aid can indirectly affect the intention of use as an important condition to increase adherence to reporting guidelines.

Half of the sample had never used any reporting guideline before participating in this study. However, after being exposed to the two tools, more than two-thirds of the sample answered that the Writing Aid was their preferred method of use. It is important to note that preferences might not lead to intention of use and actual use. This study sheds light on subjective and objective knowledge as important prerequisites for the application of reporting guidelines. The findings support Shamseer et al’s recommendation for a more active approach to improve reporting guidelines implementation, targeting the knowledge, beliefs, education and motivations of authors.34 Earlier introduction of reporting guidelines as a Writing Aid could become a formative process, where researchers are continuously exposed to and reminded of the content and use of the reporting guidelines items, leading to more complete research papers. Moreover, writing is an iterative process, thus repeated exposures to guidelines within and throughout the process may result in the greatest benefits to adherence. In addition, a digital ecosystem of software is increasingly being used to do research (eg, reference management software), and integrated tools such as the Writing Aid can be of added value. Furthermore, a user friendly system of applying the reporting guidelines can enhance self-efficacy towards their use.35 Authors are generally unaware of the value of reporting guidelines and those responding to peer reviewers have problems adhering to reporting guidelines.10 Thus, aligning education efforts to integrate reporting guidelines into the workflow, as educational tools, could be the first step. A holistic system approach and support (universities, professors, peer reviewers, journals) is needed to encourage the use, and uptake of writing aids.35

Our study had several strengths. We applied the tools within an approximation of a real life setting with participants who were in the process of writing-up personal research findings. Second, the tool works offline, which allowed us to have participants from a variety of settings, including countries with poor internet connectivity (ie, Ethiopia). Third, we accommodated a variety of topics and research designs. Lastly, we assessed the subjective and objective knowledge of the participants at baseline. With a new version, the Writing Aid software could incorporate more reporting guidelines. Furthermore, the Writing Aid software is open access and constructive contributions to improve the software are welcomed.

Our study had some limitations. First, to minimise dropouts, we did not include a washout period and conducted both interventions on the same manuscript in one session. The fact that half of our sample was not exposed to reporting guidelines before could have increased the chances of treatment period interaction, including a ceiling effect. SEM, which was conducted as a secondary analysis, was potentially underpowered. A larger sample size could have increased the power of the study, the statistical significance and the bias in the parameter estimates used in the SEM.36 Second, participants were asked to test both tools on the same manuscript in a testing session that lasted 1 hour. The length of exposure is not representative of the whole writing procedure, which is a lengthy process that contains several iterations between coauthors. Third, we did not assess actual reporting completeness or correct filling of the checklist. Most manuscripts were still in draft form and were not collected as a part of the study. Fourth, purposive sampling was used. The majority (80%) of the participants were PhD students, which might be unrepresentative for other authors. Further assessment in authors with more seniority is required. Fifth, intention of use, perceived ease of use and perceived usefulness were all collected at the same time, thus not allowing enough time for participants to experiment with the tools and assess the intention of use and actual use correctly. We consider the present study as a first step to assess the usefulness of our Writing Aid, whereas assessing reporting completeness was neither relevant, nor realistic at this stage.

Conclusions

The results of our study encourage a follow-up randomised controlled study with a longer exposure time and washout period. This will offer the possibility to further explore the potential applicability of our Writing Aid to enhance reporting guideline adherence. The findings of this study are encouraging for further product development and testing in a more representative sample of researchers.