Literature DB >> 29470789

Evaluation of Psychometric and Linguistic Properties of the Italian Adolescent Pain Assessment Scales: A Systematic Review.

Flavio Marti1, Antonella Paladini2, Giustino Varrassi2,3, Roberto Latina4.   

Abstract

INTRODUCTION: Adolescents' pain experiences are complex and multidimensional, and evaluating pain only from a sensory and affective point of view may be in many instances limiting and inadequate; this is the reason why it is of paramount importance to identify the tools which can better assess the pain experienced by young patients. A person-oriented approach is highly encouraged, as it may better investigate the cognitive and behavioral development typical of this age group. The aim of this review paper is to describe the available tools which are able to adequately assess pain intensity in adolescents, in particular those validated in Italian.
METHODS: We conducted a systematic review using four databases: CINAHL, PsycINFO, PubMed and Cochrane, and selected all the articles published between January 1970 and November 2017. We selected all the papers reporting the validation process of pain assessment tools specifically tailored for adolescent patients (age range 10-18 years) and based on psychometric and linguistic parameters, and focused especially on the tools available in Italian and able to measure acute and chronic pain.
RESULTS: The results of our investigation have revealed the existence of 40 eligible tools, 17 of which are monodimensional and the remaining 23 multidimensional, more specifically tailored to assess both acute and chronic pain. Some of the instruments (26) were self-reports while others were classified as behavioral (13) and/or mixed. Only one tool turned out to be suitable for fragile adolescents, while six adopted a person-oriented approach that better emphasized the cognitive and behavioral process typical of the adolescent population. None of them has ever been validated in Italian.
CONCLUSION: Valid and reliable psychometric tools specifically organized to provide a cultural and linguistic evaluation of the patient are indeed the most recommended instruments to assess the intensity of the pain experienced by the patient, as they may provide useful information to implement a health policy aimed at identifying the best assistance programs.

Entities:  

Keywords:  Adolescent; Assessment; Italian; Pain Scale; Pain measurements; Tool

Year:  2018        PMID: 29470789      PMCID: PMC5993683          DOI: 10.1007/s40122-018-0093-x

Source DB:  PubMed          Journal:  Pain Ther


Introduction

The World Health Organization (2014) identifies adolescence as the period in human growth and development that occurs after childhood and before adulthood, from ages 10 to 19. It represents one of the critical transitions in the life span and is characterized by a tremendous pace in growth and by important biological changes. Although the biological determinants of adolescence are fairly universal, the duration and defining characteristics of this period may vary across time, cultures, and socioeconomic situations [1]. During each growing phase in a child’s life, pain is experienced in different ways, according to the child’s age and his evolutionary, psychological and biological development [2]. The child’s communicative capacity, comprehension skills and past painful experiences may affect the way of perceiving pain throughout his whole life. Many children and adolescents with repeated acute episodes of nociceptive pain develop chronic pain that increases the risk of pain, as well as physical and psychiatric disorders in adulthood [3]. Patients aged 0–18 make up about 22% of admissions to emergency departments in the USA; among the symptoms for seeking hospital care, reporting of pain [4] seems to be quite common, and the prevalence of self-reporting increases proportionally with age. A painful event may vary from the typical everyday pain experienced by a child (i.e., bumps and bruises) [5] to more serious causes that require medical care or admission to hospital [6]. Pain is a subjective and multidimensional experience [7], which needs to be carefully assessed when it occurs in adolescents, as patients falling within this particular age group tend to minimize or even to deny pain to their parents and friends [8]. The cognitive and psychological development of adolescents gives pain a multidimensional perspective not limited to the sensory experience; sometimes the cause of pain itself may influence the way people individually respond to it [9]. Pain in some adolescents upsets their daily life, in that the intense fear of pain and disability emerge as particularly important, also influencing the lives of their parents that report significant distress and changes in their roles [10]. In addition, health professionals may be influenced by false and outdated beliefs which lead them to misinterpret the way pediatric patients feel pain, influencing therefore their capacity to properly assess its real intensity [11]. Misinterpreting results and underestimating pain may lead to an inadequate management of both acute and chronic pain [12]. As highlighted in a recent Italian survey, we observed that Italian nurses lack an adequate preparation to properly assess pain, which may account for the general low prescriptive appropriateness [13]. Therefore, the health personnel dealing with patients belonging to this specific age group must be aware of their particular bio-psychological characteristics and should find the most suitable assessment tools normally difficult to apply in clinical practice [14]. Of note, a lack of pain education in nursing and medical schools is the major cause for underestimating chronic pain in adolescents, especially from a diagnostic point of view [15], as chronic pain is normally attributed and identified only with older patients. It is estimated that about 25% of pediatric patients suffer from chronic pain [16] and these findings confirm that, differently than expected, it is a common complaint in childhood and adolescence, the most affected age group represented by adolescents between 12 and 15 years of age [17]. Chronic and acute pain assessment tools to measure the intensity of pain in adolescent patients are countless and may be classified as self-reports, behavioral observations or physiologic measures. Assessments that use multiple measures (behavioral and physiologic) and assess different aspects of the pain experience (i.e., intensity, location, pattern, context, and meaning) may result in more accurate appraisals of pediatric pain experiences [8]. Self-reporting methods are considered the gold standard for the assessment of pain, both in adolescents and younger patients [11], and may use verbal and non-verbal tools; however, only children who have attained a certain degree of cognitive ability are able to provide information this way. The capacity to self-report increases proportionally to the age of the patient and may be further enriched by life experiences typical of patients classified as adolescents [18]. Preverbal patients or those with serious cognitive impairments cannot communicate their pain in words [19]. In the absence of self-reporting, behavioral and physiologic parameters have to be used to infer pain. Behavioral indicators of pain include facial expressions, weeping, gross motor movements, changes in behavioral state and patterns [20]. In some instances, behavioral and self-report tools may be associated to physiological scales that may measure the pediatric patient’s response to stressful events by using parameters such as heart rate, blood pressure and sweating of the hands [8]. The available literature reports the existence of multidimensional scales, which simultaneously use self-report, behavioral and physiological variables to assess frail pediatric patients [21]. Given the high number of pain assessment tools reported in literature that can be specifically applied to adolescent patients, the aim of this study was to summarize and describe their most remarkable psychometric and linguistic features. This may help health practitioners who need to be advised on how to apply such tools in clinical practice. Particular attention was paid to all the tools validated in Italian, which are therefore fit to be used in national clinical settings.

Methods

Research Strategies

This systematic review was conducted according to PRISMA guidelines, using four databases: CINAHL Plus with full text, PsycINFO, PubMed and Cochrane. Information was retrieved by using the following key words: pain measurement, adolescent, pain scale. A search string that could efficiently retrieve studies on (adolescent) AND (pain OR analgesia) AND (scale OR assessment) was developed and all the articles published from January 1970 to November 2017 were selected. Secondly, the results were further filtered by including patients between 10 and 18 years of age. A more accurate selection was possible thanks to the snowballing sampling strategy, which implied the careful reading of all the eligible articles, checking if the included articles cited any other relevant article that respect our inclusion criteria. We retrieved those articles and continued this process until the absence of other relevant articles. The snowballing strategy allowed us to include the pain assessment tools which had been missed by the electronic database research strategy. The same articles selected by two or more online database were included among the results only once.

Inclusion and Exclusion Criteria

We selected all the articles reporting the validation process of pain assessment tools specifically tailored for adolescent patients (age range 10–18 years) and based on psychometric and linguistic parameters. We mainly focused on the tools available in Italian and able to measure acute and chronic pain. Our revision excluded all those articles written in languages different from English and Italian, as well as those centered mostly on evaluating the best tools to be applied when deciding on a pharmacological/non-pharmacological or surgical treatment.

Analysis of the Psychometric Characteristics

Pain assessment is a multidimensional observational assessment of a patient’s experience of pain, including its characteristics and the impact it may have on daily life activities [22]. We needed valid and reliable tools to make a rigid and accurate evaluation of a parameter that could be strongly influenced by the subjective way of perceiving pain. By reliability we referred to the stability of the tool, meaning it was able to provide equivalent results after repeated administrations and regardless of the interviewer. The soundness of the tool could be demonstrated in various ways, i.e., by confirming its validity and reliability [23]. A certain tool may be considered reliable and valid if applied on a specific population, while it may be totally inadequate for another set of patients affected by different clinical conditions [7].

Data Retrieval

All the articles retrieved by the different data banks were selected according to their title and abstract and, if pertinent to our study, the full text was analyzed. During the whole selection phase, the articles were evaluated according to the inclusion/exclusion criteria previously described.

Encoding the Psychometric Characteristics

Validity (content, criterion and construct) and reliability (stability, internal consistency and equivalence) are essential to test the psychometric characteristics of pain assessment tools and are useful to subsidize the selection of trustworthy instruments that may ensure the quality of the results of studies [23]. The selected articles were then classified according to the model adopted by Law et al. [24] and already presented in a secondary study [8]. A Likert scale was used to confer different quality levels to each evaluated tool that was classified as being “excellent, satisfactory or mediocre”. A tool was considered “excellent” when it could be applied in more than two well-performed studies, “satisfactory” if used in not more than two well-performed studies and “mediocre” if applied in unsatisfactory studies or not applied at all. A researcher who wishes to use a pain assessment tool in a study or in a clinical setting usually tends to adopt the instrument that mostly satisfies the required psychometric characteristics, although he/she also refers to those studies in which that same tool has been frequently adopted. This article is based on previously conducted studies and does not contain any studies with human participants or animals performed by any of the authors.

Results

The preliminary research retrieved 1583 articles from public data banks such as PubMed, CINAHL, PsycINFO and Cochrane. Selection criteria are reported in Fig. 1. We identified 61 eligible papers which were in compliance with our inclusion/exclusion criteria. With the snowballing strategy we identified 26 more suitable articles; at the end of the review process, we were able to identify 40 tools normally used to assess pain in adolescent patients.
Fig. 1

PRISMA chart

PRISMA chart By analyzing the age range to which each tool could be applied, only six instruments (Table 1): Adolescent Pediatric Pain Tool (APPT) [25], Bath Adolescent Pain questionnaire (BAPQ) [22], Pain-related Problem List (PPL) [26], Pediatric Pain Questionnaire (PPQ) [27], Pain Stages of Change Questionnaire—Adolescent Version (PSOCQ-A) [28], and Pain Stages of Change Questionnaire—Parent Version (PSOCQ-P) [28] turned out to be suitable for assessing pain in adolescents between 10 and 18 years of age. The characteristics and strength/weakness points of the tools specifically intended for adolescents are summarized in Table 2. We also managed to identify 29 tools (Table 3) which could be applied not only to adolescents but to a wider population set (0 years to adult age). Further five tools (Table 4): Children’s Hospital of Eastern Ontario Pain Scale (CHEOPS) [29], Poker Chip Tool (HPCT) [30], Comprehensive Pain Evaluation Questionnaire Modified (CPEQ-M) [31], Face Legs, Activities, Cry, Consolability (FLACC) [32], and Observational Scale of Behavioral Distress (OSBD) [33] were instead specific for a pre-adolescent population (< 10 years), but could also be used in older patients or in mentally impaired adolescents unable to verbally assess their pain.
Table 1

Adolescent pain tools and their psychometric characteristics

ToolAgeType of PainType of measureDimensionValidityReliabilityAuthors
APPTAdolescent Pediatric Pain Tool10–17AcuteSelf-reportMultidimensionalOriginal article not retrievedSavedra et al. [25]
BAPQBath Adolescent Pain Questionnaire10–15ChronicSelf-reportMultidimensionalThe average intercorrelation values were r = 0.48 and 0.39, making the average shared variance between subscales 23 and 15%. (convergent validity).Cronbach’s α = ranging from 0.79 to 0.89. R = 0.60Eccleston et al. [22]
PSOCQ-APain Stages of Change Questionnaire (adolescent version)13–17ChronicSelf-reportMultidimensionalOriginal article not retrievedGuite et al. [28]
PSOCQ-PPain Stages of Change Questionnaire (parent version)Behavioral
PPLPain-related Problem List13–18ChronicSelf-reportMultidimensionalThe total scale and the four domains all showed convergent and divergent validity in both samples. This was also shown by t tests comparing adolescents with low pain intensity to adolescents with high pain intensity. Adolescents with high pain intensity scored higher on the total scale (t(28) = –2.15, P < 0.05).Total α = 0.82; concentration α = 0.86; mobility a α = 0.77; adaptability α = 0.71; and mood α = 0.78Weel et al. [56]
PPQPediatric Pain Questionnaire13–18ChronicSelf-reportMultidimensionalOriginal article not retrievedVarni et al. [27]

This table is meant to describe only the essential psychometric characteristics of the available pain assessment tools. In order to have a more detailed description of the reported information, the reader should refer to the original works, considering that we purposely omitted to mention the statistical tests used and their interpretation. Therefore, data reported herein are not to be considered exhaustive

Table 2

Specific characteristics of the available adolescent tools

ToolAgeTool characteristicsStrengthsWeakness

APPT

Adolescent Pediatric Pain Tool

10–17

1. The number of pain sites as a measure of pain location from marks on a body outline

2. A pain intensity score measured by a 10-cm line known as the Word Graphic Rating Scale (WGRS) and anchored by words: no pain, little, medium, large, worst possible pain

3. The number of pain quality descriptors, which yields percent scores for the sensory, affective, evaluative subscales

4. The number of temporal descriptors, which yields a percent temporal subscale

5. The percent of total pain quality and temporal descriptors as a total subscale

It measures the different dimensions of pain, including the location, intensity, quality, and temporal pattern components of the sensory dimension as well as the affective and evaluative dimensions. Potentially, it may be able to discriminate between nociceptive and neuropathic pain, and may guide the use of multimodal interventions in adolescents with a variety of pain conditions.

A major limitation in many of the studies is the small sample size in each study, and therefore one must be cautious with interpretation of the findings. The interpretability of this questionnaire could be a problem for the clinicians.

Furthermore, the studies were predominantly reflective of acute pain states, rather than the more complex, recurrent, chronic or persistent pain states.

BAPQ

Bath Adolescent Pain Questionnaire

10–15

1. Social life and relationships with people (9 items)

2. Activities and difficulties (9 items).

3. Feelings and other experiences (6 items)

4. General worries or feelings (7 items)

5. Specific worries or concerns about pain (7 items)

6. Family life (12 items)

7. Comparison between the adolescent and other people of the same age (11 items)

8. Personal consideration about pain and life (free text)

The BAPQ may offer a comprehensive way to assess the widespread deleterious impact of adolescent chronic pain in both a research and clinical setting. The questionnaire analyzes many aspects of adolescent life such as family, relation with friends, school, fear, and difficulty in daily activities. There is also a parent version that could be used to analyze the family relationships.The questionnaire is very long (62 items).

PPL

Pain-related Problem List

13–18

1. Concentration

2. Mobility

3. Adaptability

4. Mood

The PPL is a brief self-report, 18-item instrument developed to assess the impact of chronic pain in conditions commonly seen in clinical practice, in particular those that do not necessarily have an organic cause, such as headaches, abdominal pain, and back pain.Not detected

PPQ

Pediatric Pain Questionnaire

13–18

1. Information

2. Problem solving

3. Social support

4. Positivist attitude

5. Behavioral distraction

6. Cognitive distraction

7. Externalization

8. Internalization

There are also three complex scales (approach, distraction, lack of emotion).

The PPQ assesses intensity, location, and the sensory, affective, and evaluative qualities of the pain experience. Present and worst pain in the past week is each measured on a 100-mm horizontal VAS. For location, adolescents are instructed to color in the locations of the body where they are experiencing pain, using colours of their choice to indicate pain intensity. The PPQ is a well-established instrument that has been tested in various populations such as adolescents with rheumatic diseases and fibromyalgia.Not detected

PSOCQ-A

Pain Stages of Change Questionnaire (adolescent version)

13-17

Tool based on the Adult Pain Stages of Change Questionnaire. This tool consists of 3 subscales (30 items):

1. Pre-contemplation: little perceived personal responsibility for managing pain

2. Contemplation: awareness of personal responsibility for pain management, considering behavioral change

3. Action/maintenance: active involvement in learning or continued use of self-management strategies

The adolescent is asked his agreement/disagreement level for each item and to circle the answer that better reflects his condition and pain experience of a specific illness.Needs further investigations if it has to be used in a clinical setting

PSOCQ-P

Pain Stages of Change Questionnaire (parent version)

Parents

13–17

Tool based on the Adult Pain Stages of Change Questionnaire. This tool consists of 4 subscales (30 items):

1. Pre-contemplation

2. Contemplation

3. Action

4. Maintenance.

Parents are asked to provide their agreement/disagreement level for each item on behalf of the child. They will circle the answer that better reflects their child’s condition and pain experience of a specific illness.Needs further investigations if it has to be used in a clinical setting
Table 3

Pediatric pain tools also applicable to the adolescents and to the adult

ToolAgeType of painType of measureDimensionValidityReliabilityAuthors
A.A. OFPSAfrican-American Oucher Scale03–12AcuteSelf-reportMonodimensionalContent validity W statistic = 0.67, P < 0.0001, construct validity r Pearson = from 0.88 to 0.97UnclearaBeyer et al. [57]
CASColor Analog Scaleb05 +Acute, recurring, chronicSelf-reportMonodimensionalUnclear

R = 0.97

R2 values were 0.921 and 0.922 for the CAS and VAS groups, respectively.

Comparison of the average R2 values from the various models indicated that the psychophysical

relationship between children’s responses

and intensity was best described by a power function for both the CAS and VAS scales.

McGrath et al. [29]
COMFORTCOMFORT Scale0–18AcuteBehavioralMultidimensionalCongruent validity of the 'behavior' was implied by high correlations between this latent variable and VAS (0.96, 0.89, and 0.90, respectively).

Median Kappa = 0.70.

Internal consistency reliability = 0.92

Ambuel et al. [58]
CPASSChild Pain Anxiety Symptoms Scale08–18Not specifiedSelf-reportMonodimensionalTotal score on the PCS-C; r = 0.627 (P < 0.001) and anxiety sensitivity (total score r = 0.599, (P = 0.001), suggesting good preliminary construct validity

Internal consistency

(20-item: α = 0.903)

Pagè et al. [59]
CPGChronic Pain Grading11 +ChronicSelf-reportMultidimensionalOriginal article not retrievedVon Korff et al. [60]
FACEWong-Baker FACES Pain Rating Scale03 +AcuteSelf-reportMonodimensionalr = 0.93 (P < 0.001)Excellent interscale agreementWong and Baker [61]
FCTFull Cup Test07 +Not specifiedSelf-reportMonodimensionalr = 0.95α = 0.97Ergün et al. [62]
FPS-RFaces Pain Scale–Revised03 +AcuteSelf-reportMonodimensionalr = 0.93 (P < 0.001)Excellent interscale agreementHicks et al. [63]
INRSIndividualized Numeric Rating Scale06–18Not specifiedBehavioralMonodimensional

Pearson’s r > 0.60

(indicating convergent validity)

UnclearSolodiuk and Curley [64]
MPQMcGill Pain Questionnaire12 +AcuteSelf-reportMultidimensionalUnclearMelzack [35]
MPSManchester Pain Scale03 +AcuteSelf-reportMonodimensionalSpearman’s rank correlation coefficient was 0.802 (P < 0.0001; convergent validity)UnclearLyon et al. [65]
NCCPC-PVNon-communicating Children’s Pain Checklist postoperative version03–19AcuteBehavioralMonodimensional

Convergent validity: before surgery was 0.71 and after surgery was 0.72.

Sensitivity = 0.88. Specificity = 0.81

Caregivers’ internal consistency α = 0.91; Researchers’ internal consistency was satisfactory (α = 0.71).Intraclass correlations for total scores were 0.82 before surgery and 0.78 after surgery.Breau et al. [66]
NCCPC-RNon-Communicating Children’s Pain Checklist-RevisedOriginal article not retrieved
NRSNumeric Rate Scale07 +Acute, chronicSelf-reportMonodimensional

Convergent validity (r > 0.3 to 0.5)c

Discriminant validity r = 0.63 (P < 0.001)

The authors chose to evaluate the NRS in clinical practice, recognizing that there may be more variability in how the NRS was administered and that there would be a longer time interval between administrations of the two pain scales. Thus, it is possible that lower correlation coefficients are obtained.McCaffery and Beebe [67]
OUCHERThe Oucher03 +AcuteSelf-reportMonodimensionalContent validity W = 0.726, convergent validity Γ = 0.695–0.978

Test r = 0.96, P = 0.01,

retest r = 0.99, P = 0.01

Beyer et al. [68]
PBCLPain Behavior Check List03 +AcuteSelf-reportMonodimensionalCorrelations between the PBCL scale scores and the Pain Rating Index of the MPQ and the WHYMPI Pain Severity Scale range were from 0.12 to 0.35

α = 0.85

stability coefficient = 0.80

Kerns et al. [69]
PCS-CPain Catastrophizing Scale for Children08–16ChronicSelf-reportMultidimensionalThe value of pain catastrophizing in predicting validity of disability beyond age, gender and pain intensity was significant (FChange1; 36 = 4:22, P = 0:05).α = 0.81 up to 0.89Crombez et al. [70]
PEQ-CPain Experience Questionnaire—child versiond09–15ChronicSelf-reportMultidimensionalThe concordance between child and parent report was tested. There was good agreement for pain severity. Child- and parent-rated interference correlated moderately (0.37).

Internally consistent

as indicated by the respective a values: 0.88 (pain severity), 0.86 (interference), 0.78 (parental distress)

Hermann et al. [71]
PEQ-PPain Experience Questionnaire—parent version
PICICPain indicator for communicatively impaired children02 +Not specifiedBehavioralMultidimensionalr = 0.678 (P = 0.01).UnclearStallard et al. [72]
PPATPediatric Pain Assessment Tool05–16AcuteSelf-reportMultidimensionalConstruct validity = the lowered pain scores at each of the four hourly intervals confirm construct validity of tool. Convergent validity = from 0.88 to 0.98. Discriminant validity = the low relationship between the fear measure and the pain measures supports the discriminant validity of both tools, the intensity scale and the word descriptors.UnclearAbu-Saad [37]
PPPPediatric Pain Profile03–11AcuteBehavioralMultidimensionalSensitivity (1.00) and specificity (0.91) were optimized at a cut-off of 14/60. PPP has face, concurrent, and construct validity with PPP scores increasing in line with the observer’s global rating of the child’s pain and decreasing when analgesics were administered for pain.Internal consistency α from 0.75 to 0.89. Interrater reliability from 0.74 to 0.89 (intraclass correlation)Hunt et al. [54]
PQCPain Quality Cards03–16Not specifiedSelf-reportMultidimensionalUnclearRossato and Magaldi [73]
QPSQuestionnaire on Pain caused by Spasticity02–16Not specifiedMixedMultidimensionalUnavailable (qualitative study)Geister et al. [38]
SEQ-CChild Self-Efficacy Scale08–18ChronicSelf-reportMonodimensionalConstruct validity was also obtained as 23 of the 27 hypothesized correlations were confirmed.α = 0.89 for the 7 child items and 0.90 for the 7 parent itemsBursch et al. [34]
TOPSTailored Observational Pain Screene04–20Acute, chronicBehavioralMultidimensionalUnavailable (qualitative study)Parker et al. [45]
VASVisual Analog Scale05 +Acute, recurring, chronicSelf-reportMonodimensionalOriginal article not retrievedScott and Huskisson [74]
VRSVerbal rating scale04 +Acute, chronicSelf-reportMonodimensionalOriginal article not retrievedJensen et al. [75]
WGRSWord Graphic Rating Scale6 +AcuteSelf-reportMonodimensionalUnclearR = 0.91Tesler et al. [76]

Tools validated for different age groups including the adolescent population

This table is meant to describe only the essential psychometric characteristics of the available pain assessment tools. In order to have a more detailed description of the reported information, the reader should refer to the original works, considering that we purposely omitted to mention the statistical tests used and their interpretation. Therefore, data reported herein are not to be considered exhaustive

aPlease refer exclusively to the original article or contact the authors directly

bThe psychometric properties of the CAS are equivalent to those of a 165-mm horizontal VAS

cCorrelated with VAS

dAuthors suggest that PEQ-C and PEQ-P address important psychosocial facets of chronic pediatric pain. Our data suggest strong correspondence between PEQ-C/PEQ-P and the adult MPI, hence facilitating comparisons between child and adult pain patients

eIt measures the presence of pain, not its intensity

Table 4

Psychometric characteristics of the pain assessment tools applicable to the adolescent with cognitive deficits

ToolAgeType of painType of measureDimensionValidityReliabilityAuthors
FLACCFace, Legs, Activity, Cry, Consolability0–07AcuteBehavioralMonodimensionalr = 0.849a (P < 0.01)α = 0.882Merkel et al. [32]
OSBDObservational Scale of Behavioral Distress01–04AcuteBehavioralMultidimensionalOriginal article not retrievedJay et al. [33]
CHEOPSChildren’s Hospital of Eastern Ontario Pain Scale01–07AcuteBehavioralMonodimensionalOriginal article not retrievedMcGrath et al. [77]
HPCTPoker Chip Tool03–06AcuteSelf-reportMonodimensionalOriginal article not retrievedHester [78]
CPEQ-MComprehensive Pain Evaluation Questionnaire (modified)03–07ChronicSelf-reportMultidimensionalr = 0.955; (P < 0.001);

Kendall’s tau-b = 0.825,

(P < 0.001); α = 0.777;

Kruskal–Wallis = 27.4 (P < 0.001); F = 16.1

Nelli [31]

This table is meant to describe only the essential psychometric characteristics of the available pain assessment tools. In order to have a more detailed description of the reported information, the reader should refer to the original works, considering that we purposely omitted to mention the statistical tests used and their interpretation. Therefore, data reported herein are not to be considered exhaustive

a10 In correlation with COMFORT Scale

Adolescent pain tools and their psychometric characteristics This table is meant to describe only the essential psychometric characteristics of the available pain assessment tools. In order to have a more detailed description of the reported information, the reader should refer to the original works, considering that we purposely omitted to mention the statistical tests used and their interpretation. Therefore, data reported herein are not to be considered exhaustive Specific characteristics of the available adolescent tools APPT Adolescent Pediatric Pain Tool 1. The number of pain sites as a measure of pain location from marks on a body outline 2. A pain intensity score measured by a 10-cm line known as the Word Graphic Rating Scale (WGRS) and anchored by words: no pain, little, medium, large, worst possible pain 3. The number of pain quality descriptors, which yields percent scores for the sensory, affective, evaluative subscales 4. The number of temporal descriptors, which yields a percent temporal subscale 5. The percent of total pain quality and temporal descriptors as a total subscale A major limitation in many of the studies is the small sample size in each study, and therefore one must be cautious with interpretation of the findings. The interpretability of this questionnaire could be a problem for the clinicians. Furthermore, the studies were predominantly reflective of acute pain states, rather than the more complex, recurrent, chronic or persistent pain states. BAPQ Bath Adolescent Pain Questionnaire 1. Social life and relationships with people (9 items) 2. Activities and difficulties (9 items). 3. Feelings and other experiences (6 items) 4. General worries or feelings (7 items) 5. Specific worries or concerns about pain (7 items) 6. Family life (12 items) 7. Comparison between the adolescent and other people of the same age (11 items) 8. Personal consideration about pain and life (free text) PPL Pain-related Problem List 1. Concentration 2. Mobility 3. Adaptability 4. Mood PPQ Pediatric Pain Questionnaire 1. Information 2. Problem solving 3. Social support 4. Positivist attitude 5. Behavioral distraction 6. Cognitive distraction 7. Externalization 8. Internalization There are also three complex scales (approach, distraction, lack of emotion). PSOCQ-A Pain Stages of Change Questionnaire (adolescent version) Tool based on the Adult Pain Stages of Change Questionnaire. This tool consists of 3 subscales (30 items): 1. Pre-contemplation: little perceived personal responsibility for managing pain 2. Contemplation: awareness of personal responsibility for pain management, considering behavioral change 3. Action/maintenance: active involvement in learning or continued use of self-management strategies PSOCQ-P Pain Stages of Change Questionnaire (parent version) Parents 13–17 Tool based on the Adult Pain Stages of Change Questionnaire. This tool consists of 4 subscales (30 items): 1. Pre-contemplation 2. Contemplation 3. Action 4. Maintenance. Pediatric pain tools also applicable to the adolescents and to the adult R = 0.97 R2 values were 0.921 and 0.922 for the CAS and VAS groups, respectively. Comparison of the average R2 values from the various models indicated that the psychophysical relationship between children’s responses and intensity was best described by a power function for both the CAS and VAS scales. Median Kappa = 0.70. Internal consistency reliability = 0.92 Internal consistency (20-item: α = 0.903) Pearson’s r > 0.60 (indicating convergent validity) Convergent validity: before surgery was 0.71 and after surgery was 0.72. Sensitivity = 0.88. Specificity = 0.81 Convergent validity (r > 0.3 to 0.5)c Discriminant validity r = 0.63 (P < 0.001) Test r = 0.96, P = 0.01, retest r = 0.99, P = 0.01 α = 0.85 stability coefficient = 0.80 Internally consistent as indicated by the respective a values: 0.88 (pain severity), 0.86 (interference), 0.78 (parental distress) Tools validated for different age groups including the adolescent population This table is meant to describe only the essential psychometric characteristics of the available pain assessment tools. In order to have a more detailed description of the reported information, the reader should refer to the original works, considering that we purposely omitted to mention the statistical tests used and their interpretation. Therefore, data reported herein are not to be considered exhaustive aPlease refer exclusively to the original article or contact the authors directly bThe psychometric properties of the CAS are equivalent to those of a 165-mm horizontal VAS cCorrelated with VAS dAuthors suggest that PEQ-C and PEQ-P address important psychosocial facets of chronic pediatric pain. Our data suggest strong correspondence between PEQ-C/PEQ-P and the adult MPI, hence facilitating comparisons between child and adult pain patients eIt measures the presence of pain, not its intensity Psychometric characteristics of the pain assessment tools applicable to the adolescent with cognitive deficits Kendall’s tau-b = 0.825, (P < 0.001); α = 0.777; Kruskal–Wallis = 27.4 (P < 0.001); F = 16.1 This table is meant to describe only the essential psychometric characteristics of the available pain assessment tools. In order to have a more detailed description of the reported information, the reader should refer to the original works, considering that we purposely omitted to mention the statistical tests used and their interpretation. Therefore, data reported herein are not to be considered exhaustive a10 In correlation with COMFORT Scale Analyzing the dimension of the pain scales, our literature review identified 23 multidimensional scales useful to assess pain intensity in youth. The Bath Adolescent Pain Questionnaire (BAPQ) [22], for instance, evaluates social, physical and family functioning. The Child Self-Efficacy Scale (SEQ-C) [34] evaluates the adolescent’s capacity to set up friendly ties, to self-assess his school progress, to perform simple housework tasks, to take care of himself and to autonomously do his homework. The McGill Pain Questionnaire (MPQ) [35] is a tool specifically tailored to evaluate the patient’s way to manage his emotions affections. The Pain Experience Questionnaire—child version (PEQ-C) [36] monitors emotional discomfort and social support. The Pain Stages of Change Questionnaire (adolescent version; PSOCQ-A) [28] measures the youth actions and expectations, while his emotions and affections are evaluated by using the Pediatric Pain Assessment Tool (PPAT) [37]. Finally, the Pediatric Pain Questionnaire (PPQ) [27] takes into account several factors, from social support to the capacity to carry out problem-solving strategies, from the search of physical and mental distraction to externalization/internalization. Our research also found 17 monodimensional assessment tools which are mostly focused on measuring the intensity of the experienced pain. Analyzing the type of pain, we found 11 scales for the assessment of chronic pain applicable to patients ranging from 4 to 20 years of age, although only 5 turned out to be adolescent-specific. Acute pain is normally assessed with 17 scales. Five instruments are recognized as being suitable for assessing chronic and recurrent pain, and lastly seven assessment scales did not clearly specify to what kind of pain they could be favorably applied. Classifying the pain scales about the type of measurement we retrieved 13 behavioral assessment tools and 26 self-report tools. The only mixed tool that could be successfully applied to fragile children was the Questionnaire on Pain caused by Spasticity (QPS) [38]. Table 5 reports the psychometric properties rated using the criteria described by Law et al. [24].
Table 5

Psychometric properties rated using the criteria described by Law et al. [24]

ToolPsychometric propertiesToolPsychometric propertiesToolPsychometric properties
A.A. OFPSAfrican-American Oucher ScaleValidity: satisfactoryHPCTPoker Chip ToolValidity: excellentPICICPain indicator for communicatively impaired childrenValidity: mediocre
Reliability: satisfactoryReliability: excellentReliability: mediocre
APPTAdolescent Pediatric Pain ToolValidity: excellentINRSIndividualized Numeric Rating ScaleValidity: SatisfactoryPPATPediatric Pain Assessment ToolValidity: mediocre
Reliability: excellentReliability: SatisfactoryReliability: mediocre
BAPQBath Adolescent Pain QuestionnaireValidity: satisfactoryMPQMcGill Pain QuestionnaireValidity: excellentPPLPain-related Problem ListValidity: satisfactory
Reliability: satisfactoryReliability: excellentReliability: satisfactory
CASColor Analog ScaleValidity: excellentMPSManchester Pain ScaleValidity: satisfactory. Reliability: satisfactoryPPPPediatric Pain ProfileValidity: satisfactory. Reliability: satisfactory
Reliability: excellentNCCPC - PVNon-communicating Children’s Pain Checklist Postoperative versionValidity: satisfactory
CHEOPSChildren’s Hospital of Eastern Ontario Pain ScaleValidity: excellent. Reliability: excellentReliability: satisfactoryPPQPediatric Pain QuestionnaireValidity: excellent. Reliability: excellent
COMFORTCOMFORT ScaleValidity: satisfactory. Reliability: satisfactoryNCCPC-RNon-Communicating Children’s Pain Checklist-RevisedValidity: satisfactory. Reliability: satisfactoryPQCPain Quality CardsValidity: mediocre. Reliability: mediocre
CPASSChild Pain Anxiety Symptoms ScaleValidity: satisfactory. Reliability: satisfactoryNRSNumeric Rate ScaleValidity: excellent. Reliability: excellentPSOCQ-APain Stages of Change Questionnaire (adolescent version)Validity: mediocre. Reliability: mediocre
PSOCQ-PPain Stages of Change Questionnaire (parent version)Validity: mediocre
CPEQ-MComprehensive Pain Evaluation Questionnaire (modif.)Validity: mediocreOUCHERThe OucherValidity: excellentReliability: mediocre
Reliability: mediocreReliability: excellentQPSQuestionnaire on Pain caused by SpasticityValidity: mediocre. Reliability: mediocre
CPGChronic Pain GradingValidity: mediocre Reliability: mediocreOSBDObservational Scale of Behavioral DistressValidity: satisfactory. Reliability: satisfactorySEQ-CChild Self-Efficacy ScaleValidity: satisfactory. Reliability: satisfactory
FCTFull Cup TestValidity: mediocrePBCLPain Behavior Check ListValidity: mediocreTOPSTailored Observational Pain ScreenValidity: mediocre
Reliability: mediocreReliability: mediocreReliability: mediocre
FLACCFace, Legs, Activity, Cry, ConsolabilityValidity: excellent Reliability: excellentPCS-CPain Catastrophizing Scale for ChildrenValidity: excellent. Reliability: excellentVASVisual Analog ScaleValidity: excellent
Reliability: excellent
FACEWong-Baker FACES Pain Rating ScaleValidity: excellent. Reliability: excellentPEQ-CPain Experience Questionnaire—3 child versionValidity: satisfactory. Reliability: satisfactoryVRSVerbal Rating ScaleValidity: satisfactory. Reliability: satisfactory
FPS-RFaces Pain Scale–RevisedValidity: excellent. Reliability: excellentPEQ-PPain Experience Questionnaire—parent versionValidity: satisfactory. Reliability: satisfactoryWGRSWord Graphic Rating ScaleValidity: mediocre. Reliability: mediocre
Psychometric properties rated using the criteria described by Law et al. [24] Our review highlighted the lack of pain assessment tools validated in Italian and suitable for adolescent patients.

Discussion

The aim of this review was to summarize and describe psychometric and linguistic features of tool assessment for pain in adolescent patients. Adolescence is the period in human growth and development that occurs before and is different from adulthood [1], a significant psychosocial benchmark characterized by rapid growth and a unique way of experiencing pain, very likely dependent on the patients biological/psychological development, their comprehension and communicative skills and on any painful experience they may have experienced during their childhood [2]. The first step to take in order to avoid any possible underestimation of the importance of properly treating pediatric pain [39] is to use necessary and reliable tools in different clinical settings. Our review has obtained several pain assessment tools applicable to the adolescent population. Although pain assessment prefers to adopt a person-oriented approach, which better highlights the cognitive development and the behavioral attitude typical of this age group [40], our review has recognized only 6 tools that can be used on patients of an age ranging from 10 to 18 years (Table 1). Another 29 instruments, specifically reported in Table 3, can be more widely applied to a wider population of patients aged 0–18 years. Keep in mind that each assessment process has to take into account the characteristics and personal features of each evaluated subject so as to deliver a more accurate and personalized evaluation [41]. Assessment tools specifically intended for children between 0 and 7 years of age (Table 4), who in most instances are unable or only partially capable to verbally express the intensity of their pain, may also be applied to cognitively impaired pediatric patients, such as children affected by Down Syndrome, who find it very difficult to express the precise localization and intensity of their suffering [42]. These more vulnerable subjects may benefit from using body diagrams to easily identify the exact location of their pain [43]. It is important not to compromise the observers’ judgment and to avoid doing this we need to make a global assessment of the patient from a clinical point of view. It is necessary to identify the presence of pre-existing clinical conditions, comorbidity and disability, as well as consider the age, education status, communicative skills, cognitive process, ethnic/biologic/cultural aspects and any previous pain experiences [44]. This review has highlighted how multidimensional tools have outnumbered monodimensional ones, which means that particular attention has been paid to the overall assessment of the adolescent suffering from pain and the impact this has on the quality of life of the young patient. Health no longer means merely “absence of illness” but extends to a wider concept of psycho-social welfare. This new way of evaluating the patient’s symptomatology requires the use of more and more accurate tools capable of assessing the intensity of pain by analyzing additional pain-related information. Such tools are shaped on multidimensional instruments specifically applicable to the adolescent population [45]. This paper has considered 23 multidimensional pain assessment scales used to measure pain in adolescent patients; all the tools suitable for patients belonging to this age group proved to be multidimensional, meaning that pain is not limited to a sensory experience, but implies several other aspects [9]. Monodimensional scales, on the contrary, are certainly more suitable to highlight one of the most important aspects of discomfort, that is the severity of pain, defined as the fifth vital sign. The degree of severity has to be constantly monitored [46], but cannot be the only parameter used to confirm the patient’s distress [16]. Although all vital parameters may be affected by the presence of pain [47], it has been observed that significant variations do not necessarily occur only during painful procedures. Vital signs are easily accessible but represent only one of the aspects that nurses and healthcare operators should take into account when making a global pain assessment. Only five scales were suitable to measure the severity of pain in adolescent patients: BAPQ, PPQ, PPL, PSOCQ-A, and PSOCQ-P. Chronic pain is a health issue which normally characterizes the adult and geriatric population, where 50% of the subjects > 65 years of age show multiple chronic morbidity [48]; however, no age group is to be considered immune to this kind of suffering and the adolescent population seems to be one of the selected targets [16]. A study from a few years back [6] demonstrated that pediatric patients suffering from chronic pain reached a percentage of 25%. As in adults, chronic pain in younger patients is also a multidimensional condition affecting several aspects of the child’s life: social relations, education, ability to perform physical activities, and limitations which may lead to social isolation [49]. In particular, pain assessment in this specific category of patients needs to use tools capable of investigating all the parameters typical of patients belonging to this age group. The adolescent years are characterized by the maturation of emotional and cognitive abilities that provide the developing individual with capacities needed for independent functioning during adulthood, and this process affects all the patterns of logic and social significance [8]. As a matter of fact, some of the available tools enable making a comprehensive evaluation. Pain assessment includes a series of sequential steps that measure the degree of discomfort experienced by the patient; however, in assessing chronic pain, this process should also evaluate to what extent pain impacts the life of the young patient, not only from a biological point of view but also in the social and psychological spheres [16]. Pain assessment has to take into account all the changes that seriously impact the patient’s quality of life and evaluate the outcome of the administered treatments. In some instances, the reassessment process is included in the tool, as in the case of Chronic Pain Grading (CPG), Pain indicator for communicatively impaired children (PICIC), Pediatric Pain Profile (PPP), Questionnaire on Pain caused by Spasticity (QPS), which consider the systematic re-evaluation of the pain after a given time interval. The accuracy of a pain assessment tool is also given by its validation, which takes into account the linguistic and cultural characteristics of a specific territory and population. Most of the tools presently in use are published in English. Our investigation was able to identify only 10 instruments validated in Italian, Children’s Hospital of Eastern Ontario Pain Scale (CHEOPS), Face Legs Activities Cry Consolability (FLACC), Face Pain Scale (FPS), Face Pain Scale Revised (FPS-R), McGill Pain Questionnaire (MPQ), Non-communicating Children’s Pain Checklist Postoperative version (NCCPC-PV), Non-communicating Children’s Pain Checklist Revised (NCCPC-R), Numeric Rate Scale (NRS), OUCHER, and Pain Catastrophizing Scale for Children (PCS-C), which were not applicable to the adolescent population. Pain assessment tools validated in Italian may exist, although our research did not provide us with any of them. Validation of these available instruments may presently be ongoing or in press. According to the Italian National Institute for Statistics, there are about 416,7000 adolescents in Italy, totaling 14.58% of the entire population [50]; however, the epidemiology of chronic pain in this population still remains unclear. Existing literature reports that 25% of children and adolescents suffer from chronic pain [6]. The lack of pain assessment tools suitable for adolescents may have serious implications from a clinical, ethic and juridical perspective. Pain is a public health concern which is often under-reported, underestimated and untreated in a population set that will soon reach adulthood. It is important to remember that perception of pain is influenced by the level of education of the patient, but the determinants of health also include the social, economic and physical environments, as well as the person’s individual characteristics, beliefs and behaviors. All these factors inevitably influence the acceptance and outcome of medical treatment. Linguistic validation is an essential step to make a tool reliable and applicable in the cultural setting where it is used [29]. As of January 1, 2015, in Italy 1,130,946 foreign children between 0 and 18 years were regularly registered, and of these 427,014 were adolescents, all from 191 different nations, the 10 most representative being Romania, Albania, Morocco, Ukraine, the People’s Republic of China, Philippines, Moldova, India, Bangladesh and Peru. A total of 67.4% of the adolescent population in Italy is represented by these ethnic groups [50]. Although born in Italy, they are inevitably influenced by the cultural background and linguistic traditions of their parents’ native country. Therefore, pain assessment tools also have to take into account and include these important cultural aspects [51]. Linguistic validation represents a step towards a reliable applicability of the tool to patients of a given nationality; however, the different ethnic characteristics may also help to reach a more accurate assessment of pain. Our review has retrieved two “faces” of pain scales (African-American Oucher Scale and Oucher) for self-assessing the severity of pain (smiling face, unhappy face) experienced by children of different ethnic groups: Afro-American, Caucasian, Hispanic, Asiatic, and First Nations peoples (Native Americans). Obtaining an accurate self-assessment of pain is vital to gauging baseline discomfort and response to therapy; children have to be able to identify themselves in the image they see and this is why pictures representing the various ethnic groups should be used. The use of male figures alone is discouraged even if stylized, as they may prevent girls from identifying themselves with the shown picture [52]. Picture pain scales that use drawings of both sexes (boys/girls) were found only in the versions specifically validated for Asiatic and the First Nations peoples. Most of the tools which emerged from our survey (no. 26) are of the self-report type, meaning that attention is specifically focused on the importance of recognizing the subjective experience of pain [11]. The importance of subjectivity is stressed in some tools more than in others, as in the case of Individualized Numerical Rating Scale (INRS) [53], a unique and personalized pain assessment instrument tailored for a single patient. The Pediatric Pain Profile (PPP), on the contrary, is a rating scale for assessing pain in children with the help of their parents, who can understand how their child is feeling (well or unwell), thus reducing the risk of underestimating the intensity of their pain [54]. When we have to evaluate patients verbally unable to communicate, there is the need to use behavioral pain assessment tools. We found 13 tools of this kind, which measure the intensity of pain by observing the patient’s posture. As these are monodimensional tools, they are limited to measuring only the sensory aspect of pain, rather than assessing the distressful experience in its whole complexity [9], which may generate potential measurement biases if the person using the tool has not received specific training for understanding and interpreting results [12]. The Questionnaire on Pain caused by Spasticity (QPS) turned out to be the only mixed-pain assessment tool. Its peculiarity is to make three different evaluations (self-report, health care operator and parents/caregivers) in two different moments and after a 1-week interval. Although fragility is a very complex situation to evaluate, the research in this field is still very limited and underrated, probably because of the many facets that should be considered when evaluating a patient with severe cognitive and physical impairments, which may lead the operator to underestimate the true intensity of the pain [38]. Our review paper has also highlighted that pain assessment tools which are now applied to adolescent patients had originally been made to assess the intensity of pain in the adult population suffering from chronic or degenerative pathologies. It is true that chronic pain negatively affects the adolescent population and impacts on daily life activities of the youth [36]. Social relations and absence from school are undoubtedly the most affected areas in an adolescent’s life [28]. Also, when assessing the intensity of pain in this young population set, the relationship with the parents and the child’s degree of involvement have to be taken into account [36]. In particular, the Pain Experience Questionnaire—parent version (PEQ-P) analyzes the parents’ stressful thinking about their child’s pain, which inevitably impacts the child life and compromises family dynamics and social behaviors, thus emphasizing once again the concept of pain being a complex biological, psychological and social experience [55]. As for the psychometric characteristics reported in Table 1, 3, and 4, it is important to underline that the use of the model suggested by Law [53] (Table 5) for assessing the available instruments may not be completely applicable in a given clinical setting. Several are the variables which establish both the validity and reliability of a tool, which should therefore be evaluated according to the psychometric characteristics of the tool as well as to the setting in which it is used [7].

Limitations

This review presents some limitations. In the search process, papers written in languages other than English or Italian were excluded, so it is possible that relevant findings were missed. In addition, we could not retrieve the full texts of potentially pertinent papers, so these studies were not included.

Conclusion

This review indicates the following key points: The cognitive and psychological development of adolescents requires multidimensional and specific pain assessment scales. The six pain assessment tools applicable to the adolescent population are multidimensional and self-reporting; five of them have good psychometric characteristics. No tool for the evaluation of adolescent pain is translated into Italian. The lack of tools specific for the adolescent population validated in Italian poses serious limitations on the way we approach this problem because it’s of utmost importance to use the right pain assessment tools to avoid the underestimation of this condition. The use of validated scales in Italian but not specific to adolescents could be in many cases limited and inadequate. It is very important to implement the validation of adolescent’s pain tools also in the Italian language.
  59 in total

Review 1.  AGREE II: advancing guideline development, reporting and evaluation in health care.

Authors:  Melissa C Brouwers; Michelle E Kho; George P Browman; Jako S Burgers; Francoise Cluzeau; Gene Feder; Béatrice Fervers; Ian D Graham; Jeremy Grimshaw; Steven E Hanna; Peter Littlejohns; Julie Makarski; Louise Zitzelsberger
Journal:  CMAJ       Date:  2010-07-05       Impact factor: 8.262

2.  Paediatric pain: a good start.

Authors:  Patrick J McGrath
Journal:  Pain       Date:  1990-06       Impact factor: 6.961

Review 3.  The epidemiology of pain in children and adolescents: a review.

Authors:  J E Goodman; P J McGrath
Journal:  Pain       Date:  1991-09       Impact factor: 6.961

4.  Pain assessment in the patient unable to self-report: position statement with clinical practice recommendations.

Authors:  Keela Herr; Patrick J Coyne; Margo McCaffery; Renee Manworren; Sandra Merkel
Journal:  Pain Manag Nurs       Date:  2011-12       Impact factor: 1.929

Review 5.  Measurement and assessment of pain in children--a review.

Authors:  Amit A Jain; Ramakrishna Yeluri; Autar Krishen Munshi
Journal:  J Clin Pediatr Dent       Date:  2012       Impact factor: 1.065

Review 6.  Assessing pain in children with intellectual disabilities.

Authors:  Lynn M Breau; Chantel Burkitt
Journal:  Pain Res Manag       Date:  2009 Mar-Apr       Impact factor: 3.037

7.  A new analogue scale for assessing children's pain: an initial validation study.

Authors:  Patricia A McGrath; Cheryl E Seifert; Kathy N Speechley; John C Booth; Larry Stitt; Margaret C Gibson
Journal:  Pain       Date:  1996-03       Impact factor: 6.961

Review 8.  Pain assessment in older people with dementia: literature review.

Authors:  Linda McAuliffe; Rhonda Nay; Megan O'Donnell; Deirdre Fetherstonhaugh
Journal:  J Adv Nurs       Date:  2008-11-11       Impact factor: 3.187

Review 9.  Pain assessment in nonverbal children with severe cognitive impairments: the Individualized Numeric Rating Scale (INRS).

Authors:  Jean Solodiuk; Martha A Q Curley
Journal:  J Pediatr Nurs       Date:  2003-08       Impact factor: 2.145

10.  Nurses' perceptions of children's pain: a pilot study of cognitive representations.

Authors:  Catherine Van Hulle Vincent
Journal:  J Pain Symptom Manage       Date:  2007-03       Impact factor: 3.612

View more

北京卡尤迪生物科技股份有限公司 © 2022-2023.