Mixed kinds of evidence: synthesis designs and critical appraisal for systematic mixed studies reviews including qualitative, quantitative and mixed methods studies.

The present letter is to thank Drs Shaw, Larkin and Flowers for their enlightening article entitled ‘Expanding the evidence within evidence-based healthcare: thinking about the context, acceptability and feasibility of interventions’,1 and provide complementary information to your readership about synthesis designs and critical appraisal for systematic mixed studies reviews (ie, reviews that include qualitative, quantitative and mixed methods studies).

We recently published an overview of mixed methods, which describes four main types of rigorous synthesis designs for systematic mixed studies reviews (and related techniques): convergence qualitative (thematic synthesis, metanarrative synthesis, realist synthesis and critical interpretive synthesis), convergence quantitative (content analysis and Boolean analysis) and sequential (exploratory or explanatory) designs.2 Furthermore, we have proposed guidance for researchers designing, conducting and reporting systematic mixed studies reviews (http://toolkit4mixedstudiesreviews.pbworks.com).

As mentioned by Drs Shaw, Larkin and Flowers, ‘significant advancements in appraisal tools for diverse evidence have been accomplished’ (ref. 1, p.202), and they cite our Mixed Methods Appraisal Tool (MMAT) as an example.3 The MMAT contains two screening questions for all study designs, four criteria for qualitative studies or qualitative components of mixed methods studies, four criteria for each type (randomised controlled trials, non-randomised studies and quantitative descriptive studies) of quantitative study designs or quantitative components of mixed methods studies, and three criteria for mixed methods components of mixed methods study designs (http://mixedmethodsappraisaltoolpublic.pbworks.com). Compared to other tools,4 the MMAT contains specific criteria to assess the methodological quality of studies with diverse designs including mixed methods studies. In comparison to using a specific tool for each type of included study, the MMAT is efficient as it allows the use of one tool for concomitantly appraising all types of empirical studies.

The criteria for the initial MMAT were based on (1) the theoretical work of Ian Hacking on looping effects between quantitative and qualitative evidence creating mixed kinds of evidence, and (2) a review of 17 health-related systematic mixed studies reviews.3 The criteria for the current version of the MMAT were further informed by methodological experts and workshops with national and international researchers. In conclusion, the MMAT has substantive (theoretical) validity, is content validated (literature review, workshops and experts), and has been tested for efficiency and reliability.5 The MMAT is still under development, and there may be other concurrent appraisal tools available in the future.