Psychometric properties of Persian version of the research misconduct questionnaire (PRMQ).

Assessment of scientific misconduct is considered to be an increasingly important topic in medical sciences. Providing a definition for scientific research misconduct and proposing practical methods for evaluating and measuring it in various fields of medicine discipline are required. This study aimed at assessing the psychometric properties of Scientific Research Misconduct-Revised (SMQ-R) and Publication Pressure Questionnaires (PPQ). After translation and merging of these two questionnaires, the validity of the translated draft was evaluated by 11-member expert panel using Content Validity Index (CVI) and Content Validity Ratio (CVR). Reliability of the final questionnaire, completed by 100 participants randomly chosen from medical academic members, was assessed by calculating Cronbach's alpha coefficient. The final version was named Persian Research Misconduct Questionnaire (PRMQ) and consisted of 63 question items. The item-level content validity indices of 61 questions were above 0.79, and reliability assessment showed that 6 out of 7 subscales had alpha values higher than 0.6. Hence, PRMQ can be considered an acceptable, valid and reliable tool to measure research misconduct in biomedical sciences researches in Iran.

Introduction

What gets measured gets accomplished and improved because measurement, as performance indicator, provides information for decision-making to manage outcomes and improve results. Similarly, measurement means and tools in research misconduct follow the same rule. Commission’s report on research integrity in the United States in 1995 defined research misconduct as offenses involving misappropriation, interference, and misrepresentation (1). Despite sensitivity to scientific misconduct in all scientific fields, misconduct is especially significant in medical sciences because of misconduct’s destructive consequences (2-4). Such misconducts not only can disturb the public health status of different individuals including the patients, but also can impose extensive financial costs on healthcare systems (3, 5, 6).

Some studies suggest that the rate of serious misconduct types (e.g. data fabrication) is often lower than that of other types (7-13). Pupovac et al. conducted a survey in 2016 to assess research misconduct in the Croatian scientific community on 237 researchers where 3.8%, 9.3%, 3.8%, and 25.3% admitted plagiarism, data falsification, data fabrication, and violation of authorship rules, respectively (14). In 2014, Roberts and Favs measured the prevalence of research misconduct in academics of  the United Kingdom where more than 68% of the respondents admitted committing inappropriate co-authorship (8). Hence, measuring research misconduct is a critical issue in medical sciences that need to be addressed appropriately.

Surveys and questionnaires are among the most popular tools in assessing research misconduct in medical sciences. In 1997, Rankin and Esteves developed Scientific Misconduct Questionnaire (SMQ) to address ethical problems and scientific wrongdoings in medicine (15). Eight years later, Broome et al. revised the SMQ to accommodate the changes occurred in medical sciences. They developed 68 closed-choice items and 12 open-ended items in the revised version, called Scientific Misconduct Questionnaire-Revised (SMQ-R), and used it in national surveys after assessing its psychometric characteristics (16, 17). Furthermore, the 14-item Publication Pressure Questionnaire (PPQ) developed by Tijdink et al. in 2014 was highly cited, which PPQ mainly emphasized on the considerable impact of different academic communities on scientific productivity (18).

Several studies aimed at assessing and measuring scientific misconduct in Iran. Khadem-Rezaiyan and Dadgarmoghaddam conducted a cross-sectional study using a 4-item questionnaire to evaluate the prevalence of major types of research misconducts in Iran in 2015 (19). Hadji et al. in 2016 studied the prevalence of various publication misconduct among Iranian authors in the medical field using a 5-item questionnaire (20). Review of current literature suggests that further means and tools are required to comprehensively inspect research misconduct among the Iranian medical researchers (19-23).

In this study, SMQ-R was translated from English to Persian because of its comprehensive approach in addressing topics related to research misconduct: perception of workplace environment, prevalence of scientific misconduct, awareness and beliefs about misconduct, reporting of research misconduct, and behavioral influences of research misconduct. Academic organizations in Iran impose pressure on students and faculty members to publish more articles, and PPQ covers various aspects of publication pressure. Hence, PPQ was also translated from English to Persian in this study. Several studies discussed policies imposing such pressure in Iran, showing the fundamental importance and necessity of formulating appropriate regulations in this regard (23-25). The contributions of this work are the translation of SMQ-R and PPQ to Persian, merging them, and evaluation of psychometric properties of the Persian version.

Method

This study was focused on quality assessment of SMQ-R and PPQ’s translated version. To ensure credibility, steps involved in this study was reassessed according to Streiner and Kottner’s protocol (26). Figure 1 demonstrates the methodology of this study in a workflow diagram. Initially, permissions for translation of the questionnaires were obtained through email from the developers of the original PPQ and SMQ-Rz. Two authors of this study, who were acquainted with academic writing style with acceptable English language skills, independently translated the questionnaires. These two drafts were integrated and revised in a meeting involving the authors and a methodologist. In addition, three meetings were held to develop the preliminary draft in the forward translation process. Furthermore, to evaluate the translated version’s conceptualization and relevancy, two peer academic researchers (native Persian speaking academic staff with a competent knowledge of English language) independently translated the Persian version back to English.

Figure 1

Development Workflow of Persian

Research Misconduct Questionnaire (PRMQ)

After the forward and backward translation procedures, an expert committee was formed to further assess the preliminary PRMQ’s semantic, grammatical, and lexical accuracy. The committee consisted of the following members: forward translation authors (two authors), backward translation authors (two peer academic researchers), a methodologist, and two independent colleague researchers. Moreover, face validity of the tool was qualitatively evaluated in a meeting to ensure that the questions cover the intended topics and measures the phenomenon to be measured.

Cognitive debriefing was conducted as follows: (i) selecting several volunteers from the academic staff of medical universities, (ii) asking them to complete the preliminary PRMQ questionnaires, and (iii) requesting them to leave comments regarding modifications that make the questionnaire more coherent. Fifty-four participants were selected from vice-chancellors of medical universities from all-around Iran, which attended a meeting held in the city of Rasht in 2019. The volunteers were asked to complete consent forms covering the following topics: (i) their autonomy in participation; (ii) their participation did not cause any harms, (iii) no compensations or rewards were provided for participation, and (iv) maintaining confidentiality of the participants’ information and answers. After responding to the questionnaire’s items, the participants were enquired about their understanding of items and their thoughts regarding their answers. This process ensured that this adapted and localized PRMQ functions properly when applied in the target discipline. Moreover, the committee panel evaluated participants’ answers to the preliminary PRMQ.

The content validity assessment was conducted in two stages. Initially, an expert panel was formed consisting of 11 independent scientists from different disciplines of medical sciences including epidemiology, social medicine, basic sciences and clinical sciences. The inclusion criteria for the expert panel were their experience in research conduct and research management, indicative of dexterity in their specific research disciplines. Information about the study and its goals was provided to the expert panel members, and their answers to the questions were assured to be kept confidential and anonymous.

Waltz and Bausell’s index was used to calculate Content Validity Index (CVI), and Lawshe’s index was utilized to assess the Content Validity Ratio (CVR) (27, 28). The expert panel members were requested to specify whether an item is “relevant” (CVI test), “clear” (CVI test), “simple” (CVI test), “essential” (CVR test), “useful but not essential” (CVR test), and “not necessary” (CVR test). To do so, the experts were requested to score each item of the questionnaire from 1 to 4 indicative of the degree of relevance, clarity, or simplicity: 1 (strongly disagree), 2 (disagree), 3 (agree) or 4 (strongly agree). Additionally, the members were asked to rank each item from 1 to 3 depending on the degree of its necessity: 1 (not necessary), 2 (useful but not essential), or 3 (essential). In calculating item-level-CVI index (I-CVI), an approved item had a score of 3 or 4 from all expert panel members. Total CVI was the sum of CVIs of all items, and a scale-level-CVI (S-CVI) was the ratio of the total CVI to the total number of items.

For any item with CVI and CVR lower than 0.79 and 0.59 respectively, the expert committee decided whether to include the item in the final questionnaire or not. The thresholds were not set as absolute cut-off points. Instead, they guided the expert committee to discuss their viewpoints regarding the contribution of each item in quantification of research misconduct. Since many preliminary PRMQ’s items were not valid based on opinions of the committee, a second round of content validity analysis was run for the final PRMQ with the same committee one month after this first assessment.

The final version of PRMQ was sent to 100 participants randomly chosen from medical academic members in Iran in 2019. Cronbach’s alpha test was used to analyze the internal consistency of PRMQ.

Gathering and cleaning of the raw data was performed using Microsoft Excel (2017). Then, statistical analyses on the data were done using IBM SPSS Statistics for Windows, version 24 (IBM Corp., Armonk, N.Y., USA).

Results

The comparisons showed that some items of the first draft of PRMQ needed to be removed, revised, re-worded, or re-emphasized. In the “demographic information” subscale, questions 6, 7, 9, 10, 11, 12, 13, 17, and 18 were repetitive, and hence removed. In addition, question 8, “Do you hold certification in clinical research?”, was replaced with “What is your current H-index?”. In the “prevalence of scientific misconduct” subscale, item 35, “International protocol violations related to procedures” was replaced with “Data fabrication”.

The preliminary PRMQ with 82 items was evaluated by the expert panel. To assess content validity, CVIs were calculated representing the relevance, clarity, and simplicity of all items of PRMQ except items related to the demographic information. The S-CVIs of relevancy, clarity, and simplicity were 0.797, 0.791, and 0.794, respectively. Table 1 demonstrates the detailed results of content validity assessment.

Table 1

Final PRMQ’s Content Validity Results

Q#	Description	I-CVI-R		I-CVI-C	I-CVI-S	CVR
11	Researcher competitiveness	0.91	0.91		0.91		0.63
12	Pressure on researchers to obtain tenure	1.00	1.00		1.00		1
13	Pressure on researcher to obtain external funding	1.00	1.00		1.00		0.81
14	Severity of penalties for scientific misconduct	0.91	0.91		0.91		1
15	Chances of getting caught for scientific misconduct if it occurs	1.00	1.00		1.00		0.81
16	Researchers' understanding of rules and procedures related to scientific misconduct	1.00	1.00		1.00		1
17	Your own understanding of rules and procedures related to scientific misconduct	1.00	1.00		1.00		0.81
18	Other research staff's understanding of rules and procedures related to scientific misconduct	1.00	0.91		0.91		0.63
19	Researchers' support of rules and procedures related to scientific misconduct	0.91	0.91		0.91		0.63
20	Research coordinators' support of rules and procedures related to scientific misconduct	1.00	1.00		0.82		1
21	Other research staff's support of rules and procedures related to scientific misconduct	1.00	1.00		1.00		0.63
Q #	Description	I-CVI-R		I-CVI-C	I-CVI-S	CVR
22	The effectiveness of your organization's rules and procedures for reducing scientific misconduct	0.91	0.91		0.91		1
23	Plagiarism	0.91	0.82		0.91		0.81
24	Falsifying data	0.82	1.00		1.00		0.81
25	Data fabrication	0.91	0.91		0.91		1
26	Intentional protocol violations related to subject enrollment	1.00	0.92		0.91		1
27	Coercion of potential subjects	0.91	0.91		0.82		0.81
28	Selective dropping of data from "outlier" cases	0.91	0.91		0.91		0.63
29	Disagreements about authorship	1.00	0.82		1.00		0.81
30	Pressures from study sponsor to engage in unethical practices	1.00	0.91		0.91		0.63
31	In your work environment, how often have you been aware that a researcher engaged in scientific misconduct during the past year?	1.00	0.91		1.00		0.81
32	In your work environment, how often have you been aware that a research coordinator or the other personnel engaged in scientific misconduct during the past year?	1.00	0.91		0.82		1
Q#	Description	I-CVI-R		I-CVI-C	I-CVI-S	CVR
33	How did you learn about the instances of scientific misconduct you are aware of (check all that apply)?	0.82	0.91		0.82		1
34	What do you think a typical research coordinator in your area would do if they knew a principal or co-researcher violated accepted rules for research integrity on a research project or assignment?	0.91	0.82		0.82		0.63
35	What do you think a typical research coordinator in your area would do if they knew a research team member or a staff member violated accepted rules for research integrity on a research project or assignment?	0.91	0.82		0.91		0.81
36	If someone engaged in scientific misconduct and was reported to your institutional authorities, how likely do you think it is that they would be disciplined?	0.91	0.91		0.91		0.63
37	I am concerned about the amount of misconduct	1.00	0.91		1.00		0.63
38	I think the responsibility for the scientific integrity of a study lies with the principal researcher only	0.91	0.91		0.91		1
Q #	Description	I-CVI-R		I-CVI-C	I-CVI-S	CVR
39	All professional education programs should include information about standards of research ethics	0.82	0.82		0.91		0.63
40	I feel uncomfortable talking with research coordinators and other research personnel about their ethical behavior	0.82	0.82		0.91		1
41	Dishonesty and misrepresentation of data is common in society and doesn't really hurt anybody	0.73	0.91		1.00		1
42	Pressure for tenure.	0.82	1.00		0.91		0.63
43	Pressure for external funding	1.00	0.91		0.91		0.81
44	Need for recognition	0.82	0.91		0.82		0.81
45	Need for publication	1.00	1.00		0.91		1
46	Unclear definitions of misconduct	1.00	0.91		1.00		0.81
47	Insufficient censure for misconduct	0.91	0.91		0.91		0.63
48	Financial conflict of interest	0.82	0.91		0.82		0.81
49	Number of protocols research coordinator is responsible for	0.91	0.82		0.91		1
50	Without publication pressure, your scientific output would be of higher quality	0.91	0.91		0.91		0.63
51	Your scientific publications contribute to better (future) medical care	0.91	0.91		0.91		1
Q #	Description	I-CVI-R		I-CVI-C	I-CVI-S	CVR
52	You experience your colleagues’ assessment of you on the basis of your publications as stressful	1.00	0.82		0.91		0.81
53	You experience the publication criteria formulated by your university for your appointment or re-appointment as professor as a stimulus	1.00	1.00		0.82		0.63
54	Publication pressure puts pressure on relationships with fellow researchers	0.91	0.91		0.91		1
55	You suspect that publication pressure leads some colleagues (whether intentionally or not) to color data	1.00	0.91		0.91		0.63
56	The validity of medical world literature is increased by the publication pressure in scientific centers	0.91	0.82		1.00		0.81
57	Publication pressure leads to serious worldwide doubts about the validity of research results	1.00	0.91		0.82		1
58	In your opinion, the pressure to publish scientific articles has become too high	0.91	0.91		1.00		0.81
59	The competitive scientific climate stimulates me to publish more	0.91	0.82		0.82		0.63
60	Your colleagues judge you mainly on the basis of your publications.	0.91	0.91		0.91		1
Q #	Description	I-CVI-R		I-CVI-C	I-CVI-S	CVR
61	Fellow scientists maintain their clinical and teaching skills well, despite publication pressure	1.00	0.91		0.91		1
62	You cannot confide innovative research proposals to your colleagues	0.73	0.73		0.64		0.81
63	Publication pressure harms science	0.91	0.91		0.91		0.63

PRMQ: Persian Research Misconduct Questionnaire

Q # : Question number in the final PRMQ

I-CVI-R: Item-level Content Validity Index for Relevancy

I-CVI-C: Item-level Content Validity Index for Clarity

I-CVI-S: Item-level Content Validity Index for Simplicity

CVR: Content Validity Ratio

In the “prevalence of scientific misconduct” subscale, questions 37 and 39 were omitted due to I-CVIs lower than cutoff points. Moreover, in the “behavioral influences on scientific misconduct” subscale, questions 60 to 67 were removed. Moreover, the section “actual experiences with scientific misconduct” was eliminated to reduce structural heterogeneity within the PRMQ and raise its comprehensibility level. The final PRMQ, available at

https://forms.gle/LGp85PqvFQjbXzcUA, consisting of 63 items, was elicited from the preliminary version. From these 63 questions, 10 were about demographic information, and the validity of all subscales except the demographic information were rechecked. After changing some words for several items to make them more fluent and comprehensible, content validity was re-assessed by the panel through an online survey. Scale-level content validity indices were 0.929, 0.909, and 0.912 for relevancy, clarity, and simplicity of the items, respectively.

In this step, the participants were 71 males and 29 females with age ranged between 33 to 64 and a mean of 47.97 years (SD = 7.43). From education level perspective, 81% had a doctoral degree and 19% held a master’s degree or a clinical specialty degree. Additionally, the median of the participants’ H-indices was 13. Ninety-one percent of the respondents replied “high” or “very high” to the item enquiring about the rate of research misconduct in their workplaces. Table 2 depicts the results of reliability assessment.

Table 2

Final PRMQ’s reliability assessment results

Section	Item Numbers		Cronbach’s alpha
Perception of workplace Environment	12	0.648
Prevalence of scientific misconduct	8	0.877
Awareness of research misconduct	3*	0.786
Reporting research misconduct	3	0.743
Beliefs about research misconduct	5**	-
Behavioral influences	8	0.612
Publication pressure	14***	0.807

Since an item in this subscale has multiple response option format, two out of a three items were considered in reliability assessment.

This section did was not considered a distinct subscale because the items measured distinct constructs. Hence, we did not report an alpha coefficient for this section.

All items in this section assembled to be a unified construct, and only one alpha coefficient was reported for all.

Discussion

The prevalence of research misconducts by Iranian researchers in the field of medicine has been expressed by several medical journals (21). Other studies also emphasized on the high rate of scientific misconduct in Iran's health system, which is due to the policies of this system regarding medical research and misleading orientations stimulating misconduct in medical research (19, 24, 29). Hence, this work aimed at translating, assessing, and validating the psychometric properties of the SMQ-R and PPQ as valid and reliable tools to be used in researches in Persian. Translating these two from English into Persian required cultural and conceptual adaptations achieved by several amendments to the translated and merged drafts. A pilot test involving cognitive debriefing was run to ensure that the Persian version covered the intended topics equivalent to the English version. To validate preliminary PRMQ, consisting of 82 items, both qualitative and quantitative methods were employed to approve face validity and content validity. A large number of participants were needed to assess the construct validity of PRMQ considering its relatively high number of questions, and hence construct validity was not assessed in this work and left for in future studies.

All changes made to the preliminary PRMQ were such that they had minimal impact on the integration of SMQ-R and PPQ. Amendments to the preliminary PRMQ included deleting of 19 items and rewording of several items. In the final PRMQ, all items except two had I-CVIs greater than 0.79. One of these two items was the last question in the “attitudes and beliefs about scientific misconduct” section with the following description: “Dishonesty and misrepresentation of data is common in society and does not really hurt anybody.” Another item having I-CVI of 0.73 was in the “publication pressure” section with the following description: “I cannot confide innovative research proposals to my colleagues.” The expert panel found these two items irrelevant; however, since both cases can have many interpretations, they were not removed from the final PRMQ. The item-level and scale-level content validity indices were indicative of significant changes in the final PRMQ compared to the preliminary PRMQ, showing the efficiency of the translation and the proofreading procedures. In the preliminary PRMQ, most items were considered not “necessary”. On the contrary, in the final PRMQ, given that CVRs were all above 0.59, all items were essential sub-constructs according to the expert panel.

Two subscales, “perceptions of workplace environment” and “behavioral Influences”, both depicted alpha value between 0.6 and 0.7, or an acceptable threshold. Two other sections, of the SMQ-R, “awareness of misconduct” and “reporting research misconduct”, showed a higher internal consistency with alpha value between 0.7 and 0.8. Another section of SMQ-R, “prevalence of scientific misconduct”, and all items of the PPQ showed alpha value between 0.8 and 0.9, or an excellent score. Only one subscale in the SMQ-R showed inconsistent results, “attitudes and beliefs about scientific misconduct” section. This inconsistency was in accordance with the findings of Broome et al. stating that items of this subscale measured separate constructs, and hence a meaningful consistency as a whole was not achievable for this section (16).

Khajedaluee et al. conducted a study in 2019 in Iran resulted in the development and psychometric assessment of a 75-item questionnaire consisting of three distinct questionnaires: SMQ-R, attitude towards plagiarism questionnaire, as well as positive and negative attitude and subjective norms towards plagiarism questionnaire. This study, similar to our work, considered SMQ-R to be baseline. Content validity was measured using CVIs and CVRs, and reliability was assessed using Cronbach’s alpha coefficient. Their results showed acceptable CVIs (above 0.79) and CVRs (above 0.75) for all the items, and overall alpha was 0.77                                    (30).

In the last stage of our study, 69% and 22% of 100 participants reported “high” and “very high” rates of misconduct in their workplaces, respectively. Only 9% of 100 participants reported “low” rate, highlighting the importance of measuring misconduct as well as developing or reinforcing regulations for researches by medical science academics.

In a survey conducted by Okonta and Rossouw, 68.9% of 132 researchers from Nigeria admitted committing at least eight listed forms of research misconduct, and 42% of these researchers had falsified data or committed plagiarism (31). Another national survey by Saberi-Karimian et al. in 2018 reported that 43% of academics engaged in at least one misconduct over the past three years (23). Our results, in line with these findings, highlighted the prevalence of research misconducts. Our questionnaire tool can help future research in assessing the extent of scientific misconduct in medical sciences researches. Using this tool, medical universities, academic centers, and institutions can measure misconducts to develop or reinforce regulations for their future researches.

Conclusion

Persian Research Misconduct Questionnaire (PRMQ), Persian version of research misconduct questionnaire, can be a valid and reliable tool in assessing research misconducts in the medical sciences. This tool can be approved and utilized by healthcare’s policy-making officials and managers as well as regulatory officers to measure research misconducts in the affiliated departments and centers.