Scientific integrity in research methods.

Introduction

Recent cases of research misconduct have prompted psychologists to suggest that there is too much vulnerability in the research process (e.g., Simmons et al., 2011; Pashler and Harris, 2012). Regardless of whether this is the case, ensuring the integrity of a discipline requires a clear understanding of what conventions and norms define the research process. In what follows, I consider the research integrity curriculum of North American psychology. In particular, I will claim that a major impediment to ensuring responsible research practices is an underspecified and understudied curriculum.

Defining curriculum

A general distinction made in the education literature is that between the explicit curriculum and implicit curriculum (e.g., Posner, 1992; Palomba and Banta, 1999). The explicit curriculum (EC) consists of the information in courses, textbooks, and workshops that are formally provided to learners. The EC contains the core concepts, norms, and values of an academic discipline. In that the content of the curriculum is clearly specified, the EC requires that learners achieve mastery of these theories, methods, and analytic skills. The implicit curriculum (IC) consists of the information that learners acquire throughout their studies that is not included in the explicit curriculum. The IC contains information that qualifies the formal curriculum such as exceptions to rules, as well as tacit knowledge or craft skills (e.g., Polanyi, 1958; Latour and Woolgar, 1979; Charlesworth et al., 1989). Craft skills can include how to select the “right” research question, how to effectively use experimental, analytic, and graphical software, and how to frame publications for acceptance. In that the IC is ad hoc, it can be considered an apprenticeship that relies on the specific knowledge and attention given to the learner by supervisors, mentors, and instructors.

The standards of psychology

Key issues in scientific integrity have been outlined by governmental and non-governmental organizations in North America. In the United States, governmental standards have been provided by the Office of Research Integrity in terms of the responsible conduct of research (Steneck, 2006). In Canada, major granting organizations have provided general standards that are conditions of receiving research funds (Tri-council, 2006). University policies often extend these general standards but are highly variable in terms of their content (e.g., Greene et al., 1985; Lind, 2005; Schoenherr and Williams-Jones, 2011). For instance, while data falsification is universally agreed upon as deviant behavior, publication practices are not addressed to the same extent. Explicit and implicit curricula are left to address these concerns.

Professional organizations such as the American Psychological Association (APA) provide standards for responsible research practices to their members. In addition to the five general principles of conduct (beneficence and non-maleficence, fidelity and responsibility, integrity, justice, and respect for people's rights and dignity), the APA also identifies 10 guidelines related to scientific integrity (Section 8.10a to 8.15; American Psychological Association, 2002/2010). These APA guidelines address the reporting of research results (fabrication and error correction), plagiarism, publication credit (inclusion criteria, contributions, and student credit), duplicate publication, data sharing (post-publication and limitations), and roles and responsibilities of reviewers. Whether, and how, these norms are presented within the curriculum is an open question that I will briefly consider below.

Undergraduate curriculum

The dearth of research on issues related to scientific integrity in psychological science can be contrasted with repeated reviews of the psychology curriculum more generally (Henry, 1938; Sanford and Fleishman, 1950; Daniel et al., 1965; Kulik, 1973; Lux and Daniel, 1978; Scheirer and Rogers, 1985; Cooney and Griffith, 1994; Perlman and McCann, 2005). For instance, Perlman and McCann (1999a) sampled the course catalogs of 400 institutions (evenly split among doctoral, comprehensive, baccalaureate, and 2-year colleges) from 1961 to 1997 to examine the courses offered by psychology departments. Courses that are provided across institutions provide insight into the family resemblance structure of the explicit scientific integrity curriculum within psychological science. If scientific integrity is viewed in terms of appropriate conduct through the planning, implementation, analysis, interpretation, and communication of research findings, identifying courses that likely contain this information provides a test for how scientific integrity is presented (Ellis, 1992). Thus, courses addressing experimental design, research methods, statistics, tests and measurement, as well as field experience represent an important starting point for queries into the scientific integrity curriculum (see Table 1).

Table 1

Percentage of institutions sampled by Perlman and McCann (.

Course related to scientific integrity	Year of university samples
	1961		1969		1975		1997		DU Δ(97–75)
	All	DU	All	DU	All	DU	All	DU
Tests and measurement	48	84	48	81	51	79	51	68	−11
Statistics	36	74	43	75	46	70	48	72	+2
Experimental	45	84	45	73	39	50	44	57	+7
Field experience	–	–	–	–	30	32	30	44	+12
Research methods	–	–	–	–	42	34	42	58	+24

Bold numbers reflect values used to obtain difference score. Difference score reflects change in course requirements from 1975 to 1997.

Perlman and McCann's (1999a) study can be interpreted as evidence that scientific integrity-related courses were recurrent features of the psychology curriculum in most academic institutions. However, whether we consider all academic institutions that were sampled or solely doctoral institutions, it is clear that if these courses address scientific integrity issues then these issues might only be addressed in an inconsistent manner. In a follow-up study conducted by Perlman and McCann (2005), they also observed that courses that provide students with research experience were not obligatory and that there was considerable interdepartmental variability in terms of when these courses were offered. This underscores the importance of considering degree requirements.

A stronger test of the scientific integrity curriculum is to consider courses that are included in students' degree requirements. In a companion analysis of the structure of degrees in psychological science, Perlman and McCann (1999b) consider what courses were listed as degree requirements in 500 institutions. The majority of institutions listed capstone (i.e., courses that require integrating knowledge of theory and methods; 63%) and statistics (58%) as requirements while research methods courses (40%) and experimental psychology (38%) were required to a lesser extent. Other courses related to scientific integrity such as psychometrics (9%) and experimental design (7%) were degree requirements in the minority of institutions. In a comparable manner to their study of courses offered by institutions (Perlman and McCann, 1999a), Perlman and McCann (1999b) also note that degree requirements differed based on the type of institutions. For instance, whereas the majority of doctoral institutions required statistics courses (65%), comprehensive (59%) and baccalaureate institutions (49%) did so to a lesser extent. The variability in courses offered (Perlman and McCann, 1999a) and the extent to which they constitute degree requirements (Perlman and McCann, 1999b) suggest that the EC might only weakly addresses issues of scientific integrity.

Further support for curriculum variability is evidenced in the contents of research methods textbooks. Textbooks are a means to present ideal disciplinary standards in terms of core theories and evidence (e.g., Ash, 1983; Weiten and Wight, 1992; Zechmeister and Zechmeister, 2000). While research methods textbooks typically discuss issues of design (e.g., distinguishing between dependent and independent variables, participant selection, within-, or between-subjects design), scientific integrity issues (e.g., conflict of interests, data fabrication, publication practices) might not be included. Importantly, while research ethics is a near ubiquitous feature in research methods textbooks, these issues are restricted to the treatment of human and non-human participants. Although, research methods textbooks have begun to discuss misconduct, the minority mention the APA guidelines that address scientific integrity. For instance, an examination of a sample of research methods textbooks used at the author's institution (e.g., Smith and Davis, 2003; Shaughnessy et al., 2006; McBurney and White, 2010; Cozby and Rawn, 2012; Gravetter and Forzano, 2012; Leary, 2012) revealed that no textbook included all of these guidelines and that there is considerable variability in how many are discussed. While fabrication, error correction, and plagiarism were the most common forms of misconduct discussed, various aspects of publication credit and data sharing were addressed to a lesser extent. As with the EC, research methods textbooks used to support these courses do not appear to address the issues of scientific integrity in a comprehensive or consistent manner. This leaves the responsibility for scientific integrity education in the hands of individual instructors, supervisors, and mentors.

Early experiences and graduate mentorship

It might be argued that many undergraduates neither necessarily seek, nor are considered for, graduate studies. Consequently, evaluation of the undergraduate psychology curriculum might not be the best approach to examining scientific integrity issues. For instance, while graduate and post-graduate researchers are concerned with research and publication, undergraduates need not be instructed in the specific practices required to conduct research. This argument reflects specious reasoning. First, higher education is directed toward understanding a research area. Researchers must understand the basic theories, experimental methods, and analytic procedures they use directly or indirectly (e.g., Schoenherr and Hamstra, 2015). This will minimally make students better consumers of scientific knowledge. Second, both socialization and expertise development require repeated use of social conventions, declarative knowledge, and technical skill. Given graduate students' first experience with responsible research practices occurs within the undergraduate curriculum, setting an early precedent is necessary. Moreover, as Lovitts (2007) has noted in the context of the doctoral dissertation, explicit conventions are often absent or not communicated to students. Michell (1997) goes further to claim that “many psychological researchers are ignorant with respect to the methods they use…the ignorance I refer to is about the logic of methodological practices,” (p. 356). If true, this suggests graduate school is not providing adequate instruction to develop these competencies.

A likely cause is revealed when we reflect on the experiences of graduate students. Much graduate work is based on self-directed learning. While courses are offered in advanced statistical techniques (e.g., multidimensional scaling, hierarchical linear modeling, factor analysis), other aspects of research methods are alluded to in research articles or left to supervisors and mentors to explicate. As research articles are a genre and limited in the extent to which they can discuss the research process, much of the scientific integrity curriculum is necessarily implicit. It is therefore likely to vary depending on the competency and experience of faculty members, reinforcing the importance of mentorship in education in general (e.g., Bird, 2001; Paglis et al., 2006; Anderson et al., 2007) and psychology in particular (e.g., Cronan-Hillix et al., 1986; Clark et al., 2000; Forehand, 2008). Concerns over the sufficiency of this form of apprenticeship must be addressed.

Once apprenticeship is recognized as a central feature of graduate studies, the extent to which psychologists share beliefs about scientific integrity becomes a central concern. However, psychologists have been found to disagree over the priority of APA standards (Seitz and O'Neill, 1996; Hadjistavropoulos et al., 2002) and are inconsistent in their application (Williams et al., 2012). Similar results have been observed for issues of scientific integrity. Riordan et al. (1988) examined psychologists' perceptions of plagiarism and fabrication. They note that while fabrication was viewed as more detrimental to a researcher's career, psychologists believed that university action was more justified in cases of plagiarism. More recently, John et al. (2012) assessed the prevalence and perceptions of questionable research practices by psychologists. They found that the manipulation of results in an unplanned and unreported manner was a reasonably common practice while also being judged to be dishonest by psychologists.

Conclusions

Psychology is no more susceptible to disagreement over its norms than any other science (Ioannidis, 2005; De Vries et al., 2006). Consequently, variability of the undergraduate and graduate curricula suggests that a more explicit treatment of scientific integrity issues should be pursued. Despite the possibility that undergraduate statistics and research methods courses might address some of these issues in a general manner, other topics are not likely to be addressed. This appears to be reflected in the variable content of research methods textbooks. If departments are unclear as to whether this is the case, tools such as curriculum matrices (e.g., Levy et al., 1999) can be used to formally evaluate the features of their curriculum. Curriculum matrices require that faculty members identify core topics that should be covered within a curriculum and assess which courses address this information. When course information is plotted on such a grid, gaps are revealed and can then be addressed. In conjunction with the standards of professional organizations, formal policies, and guidelines can also be developed to ensure greater consistency.

Funding

This research was supported by funding from the Ottawa Hospital Research Institute.

Conflict of interest statement

The author declares that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.