Psychometrics: Essential for Valid, Reliable, and Responsive Measurement in the Development of Patient-reported Outcome Instruments in Plastic Surgery.

Sir:

We would like to take this opportunity to respond to Dr. Eric Swanson’s[1] article on the role of psychometrics in plastic surgery. As a collective, our philosophy is different from what has been described by Dr. Swanson, and we would like to illustrate some fundamental concerns we have with his article.

Throughout his article, he continually refers to “rigorous methodology,” often citing his own work.[2-5] Dr. Swanson correctly discusses techniques such as randomization, inclusion of consecutive patients, and use of prospective design as “rigorous” methodology. However, these techniques pertain to studies of intervention and the utilization of a measure in an outcome study; they do not pertain to the actual development of an outcome measure, and specifically, patient-reported outcome (PRO) instruments. An entirely different set of methodological safeguards exists for this and is detailed elsewhere.[6-8] A Food and Drug Administration scientific advisory committee[6] and an international consensus group of quality of life research experts[7,8] have set the standards for this methodology, which is rigorous in its own right. One would not use this methodology for evaluating or conducting a clinical trial; similarly, one cannot use clinical trial methodology and apply it in evaluating the development of a PRO instrument.

With regard to validity, Dr. Swanson is right in pointing out that validity is to be determined by users of the measure; however, it is erroneous to call validity a “self-serving designation that adds nothing of value to the title.” Consider the introduction of a new surgical technique. For the technique to be accepted by the surgical community, it must first be adopted, attempted, and the outcomes assessed by independent investigators to determine if indeed the technique is useful and applicable. However, for the technique to even be considered in the first place, the inventor must demonstrate some success of the technique with an initial series of patients. This analogy applies to the development of a measurement instrument as well. Some degree of initial validity, reliability, and responsiveness must be demonstrated for the measure to be adopted and further tested by other researchers. In presenting a new instrument in an article, therefore, an investigator is obliged to report this. In fact, for use of a measurement instrument in outcome studies, evidence of validity and reliability must first be demonstrated and previously published[6,9,10]; the use of unpublished instruments is notoriously fraught with bias.[11]

Whether we call an outcome instrument a survey, a test, a questionnaire, and so on is really just semantics. What we are really looking at is measurement. A ruler must be constructed in a standardized manner such that the points and numbers along the ruler actually measure what they intend to measure (validity) and does so consistently when used in the same or similar circumstances (reliability). One would have no use for a ruler where the distance between 1 and 2 cm was not the same as between 2 and 3 cm as this would undermine both validity and reliability. Similarly, when measuring abstract concepts such as physical, mental, and social well-being, what patients are able to do and how they feel, the measuring device itself is crucial, and how that device was developed underlies its validity and reliability. This leads to a discussion of psychometrics.

Psychometrics evolved from techniques of psychological measurement, and was adapted to health science research, in particular for the development of PRO instruments. PRO instruments are developed primarily from patient input through intensive qualitative methods. Psychometrics is used to evaluate the questions or items that emerge from this patient-centered input for construction of the measurement instrument. It is an integral part of the methodology for measurement instrument development adopted by national and international agencies no less than the National Institutes of Health,[12] the Food and Drug Administration,[6] and the International Society for Quality of Life Research[13] to name a few. Although the technical details (not “jargon”) of psychometrics may seem complex and foreign, reporting of these details is essential. Ten to fifteen years ago, the concepts, terminology, and statistical tests pertaining to randomized controlled trials (RCTs) and systematic reviews were foreign to most plastic surgeons. Even now, one may not understand completely the concepts of block permutations, the multiple forms of bias, statistical heterogeneity, or fixed versus random effects yet can accept that the end results of applying this rigorous methodology are studies that are the gold standards in the well-known hierarchy of evidence.[14-16]

Presentation of the methodological details in publications is essential to allow for any reader knowledgeable in the methodology to critically appraise the quality of a study. In fact, there are now widely accepted reporting guidelines for RCTs that require the presentation of numerous technical details of an RCT.[9] Similarly, presentation of the technical details of psychometric tests is essential to allow for critical appraisal and acceptance of a measurement instrument.[7,10] Furthermore, we would argue that the readership be given more credit than to assume a discussion of psychometrics is incomprehensible to them.

To ignore the importance of psychometrics in measurement development in favor of other, standard, statistical tests for the sole reason that they are more familiar and easier to understand is an oversimplification and a misguided proposal. Would one propose that an RCT be conducted attempting to control for all biases or only the ones the average reader would understand? Discounting an entire discipline on the basis of a completely inapplicable methodology and body of research lacks scientific integrity. A knowledgeable understanding of PRO instruments and how they are developed and utilized is required before educated, constructive, criticism can be provided.

Further reading for those interested in learning more about measurement and psychometrics:

A. Cano SJ, Hobart JC. The problem with health measurement. Patient Prefer Adherence 2011;5:279–290.

B. McDowell I. Measuring Health: A Guide to Rating Scales and Questionnaires. 3rd ed. New York, NY: Oxford University Press; 2006.

C. Streiner D. Health Measurement Scales: A Practical Guide to Their Development and Use. New York, NY: Oxford University Press; 2014.