Reliability studies of diagnostic methods in Indian traditional Ayurveda medicine: An overview.

Recently, a need to develop supportive new scientific evidence for contemporary Ayurveda has emerged. One of the research objectives is an assessment of the reliability of diagnoses and treatment. Reliability is a quantitative measure of consistency. It is a crucial issue in classification (such as prakriti classification), method development (pulse diagnosis), quality assurance for diagnosis and treatment and in the conduct of clinical studies. Several reliability studies are conducted in western medicine. The investigation of the reliability of traditional Chinese, Japanese and Sasang medicine diagnoses is in the formative stage. However, reliability studies in Ayurveda are in the preliminary stage. In this paper, examples are provided to illustrate relevant concepts of reliability studies of diagnostic methods and their implication in practice, education, and training. An introduction to reliability estimates and different study designs and statistical analysis is given for future studies in Ayurveda.

INTRODUCTION

Ayurveda as a traditional and holistic medicine has a sound philosophical and experiential basis.[1] Long historical use has been seen as documentation of the efficacy; however, there is lack of quantitative studies of concepts such as reliability as evaluated in the modern medicine. The term reliability means “consistency” of a measure.[2] Diagnosis is considered reliable if it gives consistent results under similar conditions. Assessment of reliability is essential because consistent diagnosis leads to consistent treatment.

In research, there are eight criteria for evaluating the patient-based outcome measures for any specific clinical trial: Appropriateness, reliability, validity, responsiveness, precision, interpretability, acceptability, and feasibility.[3] In Ayurveda, outcome measures may be the findings of physical examinations or the scores on questionnaires collecting information, for example, on body constitution (prakriti), life style, diet, and quality of life. If these outcome measures or diagnostic variables such as bodily humors/energies (dosha), tissues (dhatu), metabolic waste (mala), metabolic fire (agni), subtle life force (prana), material life force (ojas), toxicity (ama), body parts (avayava), vital body part (marma), 5 elements (mahabhoota) are to be included in the research protocol, the prerequisite is that they should be valid and reliable.[4]

In Ayurveda, diagnostic methods (such as pulse diagnosis) often rely on some degree of subjective interpretation by physicians. If the physicians cannot agree on the interpretation, the results will be of little use. Hence, reliability studies are necessary for quality assurance in the conduct of clinical studies and practice.[5]

In this paper, we provide a selected summary of the reliability studies of the physical examination commonly used in Ayurveda and different traditional medicines from Asia. It is noteworthy to reflect on the reliability studies carried out in traditional Chinese medicine (TCM), Japanese traditional medicine: Toyohari meridian therapy (TMT), and Sasang Constitutional Medicine (SCM). These traditional medicines have some similarities with Ayurveda.[16] One common finding is that diagnostic methods of all traditional medicines rely more on the physicians reading of the patient's signs and symptoms than on laboratory findings.

The scope of this paper is limited to inter and intra rater reliability for specific diagnostic methods: Pulse (nadi) diagnosis, body constitution diagnosis and tongue diagnosis. In final, future perspectives of reliability studies in Ayurveda are discussed.

METHODS

A literature review is conducted using electronic databases “PubMed” “Google Scholar” and “Scopus.” The review was conducted with an interactive strategy of combining the keywords “reliability,” “agreement,” “traditional medicine,” “alternative medicine,” “Ayurveda” “complementary medicine,” “Chinese medicine,” “Sasang medicine,” “Toyohari medicine.” Further, advanced or refined search was carried out using the key words “diagnostic methods,” “physical examination,” “pulse diagnosis,” “body constitution,” “prakriti,” and “tongue diagnosis.” Furthermore, reference lists from previous systematic reviews were browsed.[789] Articles were limited to those in the English language.

OBJECTIVES

The objective of this study was to provide information about how the reliability studies can be designed and conducted for Ayurvedic diagnostic methods. Importance of the reliability studies in practice and clinical trials will be discussed in relation to illustrative case studies.

WHAT IS RELIABILITY?

Reliability denotes consistency of a measure. Reliability provides information about the amount of error inherent in any diagnosis score or measurement, where the amount of measurement error determines the validity of the study results or scores.[5]

Reliability verses validity

Validity and reliability concepts can easily be misunderstood. Validity is analogs to accuracy. A test/instrument is valid when it measures, what it is intended to measure. The test is reliable when it produces same results under identical conditions. Thus, reliability does not denote validity [Figure 1]. For example, if a person, who weighs 50 kg steps on a weighing scale 4 times and gets readings of 45, 48, 40, and 54 kg the scale, is not reliable and if it consistently reads “45 kg” it is reliable, but not valid. If it reads “50 kg” each time, it is reliable and valid. A test that is not reliable cannot be completely valid. Measures of validity of diagnostic procedures are commonly quantifying the ability of the procedures to distinguish individuals with and without a certain disease. Basic measures for this purpose, such as sensitivity and specificity, likelihood ratios, positive, and negative predictive values are described elsewhere.[1011] More elaborate measures of validity for, e.g., psychological testing are presented in[12] It is essential that a diagnosis is reliable and valid. However, in Ayurveda, the problem with assessing validity is that there is lack of “gold standard” to compare with. E.g., for pulse diagnosis the diagnosis can only be obtained from a doctor's judgments. However, since different doctors may obtain different diagnoses, we do not know which one is the true diagnosis that all other diagnoses should be compared with.

Figure 1

Quantitative assessment of diagnostic methods/tests

TYPES OF RELIABILITY

There are several types of reliability estimates.[213] The terms “reliability” and “agreement” are often used interchangeably.[14] The two concepts are conceptually distinct. Reliability parameters are the most appropriate when the research questions concerns with the distinction of persons. However, parameters of the agreement are preferred when the aim of the study is to measure change in health status, which is often the case in clinical practice. However, similar study designs are used for examining these two concepts. Guidelines are also available for reporting reliability and agreement studies.[15]

Intra-rater reliability

It is also known as “test-retest reliability” or “repeatability.” This type of reliability is used to assess the consistency of a test across time. Intra-rater reliability testing is the process by which a measurement tool or method can be shown to give similar results when used by same raters at different time for the same group of subjects. Examples are given in Tables 1 and 2 [Figure 2].

Table 1

Reliability studies in different traditional medicines from Asia

Table 2

Reliability studies in Ayurveda

Figure 2

Intra-rater reliability

Inter-rater reliability

Inter-rater reliability or “reproducibility” denotes in clinical settings the extent to which doctors agree with each other in their diagnosis and treatment. This is assessed by allowing two or more independent doctors carry out independent assessments of the same patient. The scores are then compared to determine the consistency of the doctor's estimates. Examples are given in Tables 1 and 2 [Figure 3].

Figure 3

Inter-rater reliability

Inter-method reliability

Inter method reliability is assessed by comparing two different methods or tests [Figure 4]. When a new method is proposed its value can be assessed only by comparing with another established technique (gold standard), rather than with the true quantity being measured. It is not possible to verify that either method gives definitely correct measurement, so it is necessary to assess the degree of agreement between them. For example, an automated oscillometric blood pressure monitor was compared with the auscultator mercury sphygmomanometer.[16] This comparison between the different methods is assessed in a similar manner as for intra-rater reliability. Analysis on the agreement between two methods of clinical measurement is proposed by Bland and Altman.[17]

Figure 4

Inter-method reliability

In brief, inter and intra rater reliability estimates are used when the raters (doctors) are part of the experiments. To establish a new technique or method, inter method reliability is assessed. While dealing with forms or questionnaires/instruments, inter method reliability is termed parallel-forms reliability. A way of discovering which questions are more informative is to use two questionnaires in parallel and finalize one that is reliable. Further, reliability of each item (questions) is estimated by internal consistency reliability as explained in the next section.

Internal consistency reliability

This form of reliability is used to assess the consistency of results across items on the same test or questionnaire [Figure 5]. This can be determined by average inter-item correlation, average item total correlation, and split-half correlation.[213] The internal consistency of Sasangin diagnosis questionnaire analysis was carried out using the data collected from 423 respondents. The questionnaire consisted of a total of 229 items. Cronbach's alpha coefficient (above 0.50) showed that all the categories can be accepted as being reliable scales, meaning this questionnaire is acceptable for surveying purposes.[18]

Figure 5

Internal consistency reliability

HOW TO CONDUCT RELIABILITY STUDIES IN AYURVEDA

Intra-rater reliability can be conducted by a single or more raters on same subjects. However, reliability estimates may vary according to the time interval between repeated measures. Assessment of intra-rater reliability is difficult for some direct observable signs and symptoms, since results may be influenced by the observer's memory or attempts at consistency in observations.[19] Hence, it is necessary to keep adequate time interval in between diagnoses to avoid the carryover effect of first diagnosis. In intra-rater assessment of Sasang constitution (SC), six SCM experts diagnosed SC of 86 participants twice independently with 1 year between first and second diagnosis. The reliability was moderate.[20] The time interval of 1 year was necessary to avoid the carryover effect of the first diagnosis.

On the other hand, pulse and tongue characteristics may change within hours or a day. It makes assessment of intra-rater reliability more difficult. It is possible only if such studies are conducted in a short time to avoid possible variation in pulse or other symptoms. Furthermore, blinding and randomization is necessary to avoid carry over effect of the previous diagnosis. In an intra-rater assessment of pulse diagnosis, an Ayurvedic pulse diagnosis expert diagnosed doshaja type pulse of 17 healthy subjects twice in a random order. The trial was conducted on the same day and within a short time period. To avoid carry over effect of first diagnosis the doctor did pulse diagnosis without seeing the subjects. The reliability was moderate. Furthermore, raters should be blinded in the sense that they should not be aware of the number of participants and number of rounds.[2122]

In case of tongue diagnosis or observable signs, uses of photographs or video recording can be used for the repeated diagnosis after an adequate time interval. In TCM, tongue slides were used for the intra-rater reliability assessment.[23]

Inter-rater reliability is conducted by more than one rater on the same group of subject. In Japanese medicine, two experts independently carried out pulse, abdominal and sho diagnosis (primary and secondary patterns of disharmony) of 62 healthy subjects. Reliability was moderate for pulse diagnosis than sho diagnosis and abdominal diagnosis shown in [Table 1]. Low level of reliability on sho diagnosis suggests room for improvement.[24]

Several factors may potentially affect the reliability of clinical observations: E.g., practitioner's and patient's variability. Following factors need to be considered.

Raters

There is diversity in the practice of Ayurveda and inherent subjectivity in the clinical observation. Practitioner variability due to the practitioner's experience, education, specialization, and reliance on different traditions should be considered while interpreting results.[22] Frequently, the larger number of practitioners involved, the less likely is agreement. Moreover, it is reasonable to keep the raters blinded in the sense that they should not be aware that their diagnosis will be judged with the other raters. This will ensure that their behavior is not altered because of awareness of being observed.[25] Diagnosis should be conducted independently by each rater without communication.

Subjects

Subject's variability may have influence on the result because clinical signs and symptoms may change within some time limit such as for pulse diagnosis. It is recommended to conduct the study on the same day and the same time. Only, prakriti diagnosis can be conducted at any time as it remains unchangeable for whole life.

Diagnostic method

The degree of reliability is related to the properties of the methods or classifications that are used. Diagnostic methods exist in various forms, due to the different traditional practices. Different methods may have been adapted by different practitioners. Therefore, care should be taken in the study design that the same diagnostic method is used by all raters. Further, the specification and interpretation of diagnostic method is essential. When diagnostic methods are to be used in broader clinical contexts and daily practice, reliability should also be investigated under conditions as close as possible to the clinical daily routine.[24]

Statistical analysis

The choice of statistical method depends on the type of data (nominal, ordinal, continuous), the sampling (at random, consecutive, convenience) and treatment of random and systemic error.[5] The intra-class correlation coefficient may be useful for measuring the reliability at continuous scale.[26] Moreover, proportion of a specific agreement,[27] reliability coefficient and graphical methods are also suggested for continuous data.[17] Cohen's kappa (K), Fleiss kappa, and weighted kappa statistics are indices of reliability for use with nominal scales.[28] Kappa statistics is a recognized measure of level of agreement beyond that expected by chance alone. It gives a quantitative measure of the magnitude of agreement between observers. Possible kappa values range from + 1 (perfect agreement) via zero (no agreement above that expected by chance) to - 1 (complete disagreement). An interpretation of kappa values in terms of level of agreement is given in Table 3.[29] Examples of kappa values and their interpretation for common clinical signs are provided in Table 4.[7] These examples provide evidence about the different levels of reliability of the physical examination and common clinical signs in western medicine.

Table 3

Interpretation of kappa values by Landis and Koch scale

Table 4

Comparisons of kappa values for common clinical signs

In Ayurveda, most of the variables are nominal and categorical such as dosha, prakriti. Consequently, for Ayurveda it is relevant to understand kappa statistics. In weighted (Kw) kappa, disagreements of varying gravity (or agreements of varying degree) are weighted accordingly.[30] For example, the doctors would likely consider a diagnostic disagreement between “vata” and “kapha” to be more serious than between “vata” and “vatakapha.” If we use Cohen's kappa, it makes no distinction, implicitly treating all agreement (disagreement) equally. In an Ayurvedic pulse diagnosis study, additional interpretation of Cohen's weighted kappa statistic for analysis of categorical pulse and body constitution diagnosis was provided. For quantification of the reliability measure, weights were assigned based on the various compositions of vata, pitta, and kapha. A detailed presentation of weights based on a distance measure for pulse and body constitution diagnosis is presented in a reliabilities studies on pulse diagnosis.[21]

Comparisons of kappas across studies must be interpreted carefully because kappa values vary with prevalence.[28]

Moreover, the number of possible response categories of a test also influences kappa.[31] The kappa will be high when there are only two categories: E.g., presence or absence of a disease. As the number of categories increases, the kappa values will be smaller. Nevertheless, most of the medical literature on reliability has been reported in terms of kappa values and it remains a useful summary measure of reliability.

IMPLICATION OF RELIABILITY STUDIES

The results of the clinical trials conducted on many herbs and formulations could be improved by incorporating classical principles of Ayurveda diagnosis.[32] For this the prerequisite is that these variables should be reliable. In the following section, we will discuss body constitution, pulse, and tongue diagnosis studies from traditional medicine and their implication in research, education, and practice.

Body constitution diagnosis

Development of constitutional questionnaire

In Ayurveda, prakriti based prescription helps in enhancing the therapeutic effect as well as reduces the unwanted effects of the drug. For better results, it is important to include prakriti assessment in the clinical trial as inclusion/exclusion criteria.[33] There are few interesting studies indicating either a genetic or a biochemical basis for body constitutional types.[3435] A pilot study on development and validation of a prototype prakriti analysis tool reported that vata and pitta constructs of prakriti identification have a significant inter-rater correlation (P < 0.001 and P < 0.01), whereas kapha has less (P < 0.02) correlation. It is concluded that kapha features are required to be designed more carefully to reach better consensus.[36] Some reliability studies on SC have been carried out in SCM.[2037] Ayurvedic prakriti questionnaire includes three main categories that are vata pitta and kapha types. Internal reliability measures whether several questions that propose to measure the vata, pitta, and kapha, categories produce similar scores. Internal reliability is important while developing and evaluating a questionnaire. For example, a study on cold and heat pathologic pattern identification in TCM showed significant differences in the mean questionnaire scores between the cold and heat groups.[38] It is concluded that the questionnaire may be useful as an adjunct diagnostic tool.

In another study, the agreement between raters clinical rationale was assessed for Ayurveda diagnosis. Overall agreement on diagnoses ranged between 60% and 100% and was higher for vikriti (mean 86%) than prakriti (mean 75%) for thirteen participants.[39] Even when there was disagreement in diagnoses, the clinical rationale provided by physicians was consistent with the theoretical basis of Ayurvedic medicine. Interpretation of these studies is difficult because the study reported percentage of agreement. The result could be misleading because it does not take into account the agreement by chance.

Pulse diagnosis

Development of a standardized pulse taking procedure and importance of training

Ayurvedic pulse diagnosis is the unique and non-invasive diagnostic method that determines the state of dosha; however, this is only justifiable if pulse diagnosis yields a consistent result. Many studies in TCM and TMT reported low to very good level of reliability for pulse diagnosis. An identified reason for the low reliability of TCM pulse diagnosis was the complex and ambiguously defined TCM pulse qualities and little systematic information.[8] So, effort was taken to develop a standardized pulse taking procedure. A high level of reliability (80%) was observed when the study was conducted by developing concrete operational definitions for each of the characteristics of the pulse. However, the study reported percentage agreement instead of kappa value.[40] Thus, it is hard to judge the results.

In Ayurveda, in a double-blinded, controlled clinical trial, fifteen Ayurvedic doctors examined the dosha type of pulse of 20 (bio-medically defined) healthy subjects twice in a random order without seeing them. The doctors seem to favor different diagnoses since the proportions of ratings vary among doctors [Table 2].[22] Possibly, reliability of Ayurveda pulse diagnosis may be improved by standardizing the pulse taking procedure and by proper training.

Tongue diagnosis

Development of differential criteria on tongue diagnosis

A study on tongue diagnosis was conducted by 30 TCM practitioners. It is reported that the low level of reliability was due to inadequate operational definitions of both the tongue characteristics studied and of the inspection regions of the tongue.[23] When there is a need to develop standards for the diagnosis, it is obligatory that it should be carried out by reliable practitioners. For instance, 24 reliable raters (kappa value = 0.56) were selected from 60 rater to develop standards for judgment on the tongue coating thickness (TCT). The correlation between rater's TCT judgments and digital tongue imaging system judgments was high. Accordingly, it is proposed that thick coating of the tongue is that occupying approximately more than two-third of the tongue surface area.[41] Similarly, it will be beneficial to develop evidence based diagnostic guidelines for Ayurveda practice.

Finally, reliability studies are extended to disease diagnosis; specifically on lower back pain,[42] menopausal syndrome,[43] irritable bowel syndrome,[44] and headache[45] in TCM. Considerable agreement existed among three practitioners for the diagnosis and treatment of inflammatory polyarthritis despite Ayurvedic medicines individualized approach.[46]

FUTURE PERSPECTIVES

The diagnosis made during the Ayurvedic clinical evaluation of a patient should be consistent; Ayurvedic diagnostic variables are only justifiable if they are reproducible by different physicians for the same group of patients. Evidence of high reliability will improve the confidence among the doctors and these methods will possibly be incorporated into the clinical trials. If diagnosis is variable across different physicians, there is a need to understand the reason behind this variability. Moreover, to improve the quality and value of patient care, it is important to assess physician's performance in the clinical practice.[47] Based on the reliability results, clinical reliance should be given on reliable variables or methods.

Development of diagnostic guideline based on current scientific evidence is inevitable for contemporary Ayurveda. The study conducted by Patwardhan et al.[48] suggests that the Ayurvedic academicians are required to be trained in standard methods of research and the educational institutions should contribute in building up the evidence base for Ayurveda in the form of quality education and research as demonstrated in rheumatologic studies in Ayurveda.[49]

As observed in a few studies, if reliability is low, the key to improve the reliability is greater standardization of the most robust methods and a better understanding of the examination technique and its failings. Many studies show that training of professionals, improving the diagnostic instrument or method and a combination of both plays a significant role in greater reliability.[50] Furthermore, in TCM, a low level of reliability was observed in two subsequent studies on rheumatoid arthritis (κ value = 0.28 and κ value = 0.30 respectively).[5152] Improvement in the level of reliability (κ value = 0.73) was observed after training sessions for the practitioners from study two [Table 1].[53]

Reliability can be improved by examining more frequently or examining same patient by more than one clinician. How to best increase the number of observations depends on the nature of the variation and diagnostic method. For example, variation in the hypertension diagnosis can be overcome by measuring blood pressure on several occasions by the same clinician.[54]

In conclusion, the reliability of diagnostic methods is of concern in research, education, and clinical practice. For contemporary Ayurveda, to be recognized as a credible health-care system, there is a need for rigorous reliability studies to be performed in the future.