Subjective rating scale for discourse: Evidence from the efficacy of subjective rating scale in amnestic mild cognitive impairments.

In clinical settings, the language ability of patients with neurologic communication disorders have been measured by quantitative parameters such as the total number of words in dialogue and picture description tasks. However, this quantitative analysis requires a long period of time in order to analyze the quantitative parameters, and results can differ according to discourse tasks. The purposes of this study are to explore whether SR-D may predict the quantitative measures of discourse tasks. Forty patients with amnestic MCI and 40 normal elderly participated in the study. We gathered responses to 10 items regarding SR-D and analyzed the quantitative measures of narrative discourse through 3 discourse tasks (i.e, picture description, dialogue, procedural discourse). We found significant differences in MLTW, CIU, and SR-D scores between the 2 groups. In particular, 4 items were significantly correlated with the performance of MLTW and CIU. Sensitivity and specificity of these 4 items were 100% and 75%, respectively. In terms of economic opportunity costs, objective measures cannot be evaluated to be practical, since it is used in research rather than clinical diagnosis in general. Therefore, evaluation of discourse using a few items proven in its sensitivity and specificity could allow a wide use of such measure in not only research but also in clinical diagnosis. These findings suggest that subjective measures of narrative discourse may be valid with objective language tests to predict individual discourse performance.

Introduction

Discourse is action in a communicative setting to maintain a suitable context, gather necessary information, and make an intended response in not less than 2 sentences, and is related to a person's efficient use of cognitive resources.[ A discourse production task provides information on strengths and weaknesses of one's communicative abilities and evaluates semantic, syntactic and pragmatic information of language, thus making it possible to assess one's comprehensive communicative ability.[ Therefore, relevant studies have used discourse tasks in order to identify the comprehensive cognitive abilities of normal individuals and to distinguish the cognitive-communication impairments of individuals. It is thought that because discourse incorporates every aspect of language it is formed through complex interchanges between attention, memory, language and cognitive abilities,[ and a number of symptoms of individuals with low cognitive ability are identified in discourse.

Individuals with advanced Alzheimer's disease (AD), a typical symptom of degeneration dementia, showed a low rate of corrected information unit (CIU, %) and a low type-token ratio (TTR, %) in a picture description task.[ In addition, when quantitative measures of discourse regarding a picture description task utilizing the Boston Diagnostic Aphasia Examination (BDAE) were compared with those of a normal group, it was found out that a discourse task is effective in identifying the decreased semantic memory of an individual with AD.[ In a discourse task, normal elderly people showed a limited ability in a semantic aspect,[ but a relatively well-preserved ability in a syntactic aspect.[ However, patients with AD exhibited deficiencies in both memory and semantic aspects of language ability, even from an early stage of AD, due to temporoparietal dysfunctions, which often led to little or no information of discourse, described as “empty speech”.[

Mild cognitive impairment (MCI) is characterized by memory deficits in clinical terms. Although MCI can exhibit with a variety of symptoms, when memory loss is the predominant symptom it is termed “amnestic type of MCI (aMCI)” and is frequently seen as a preclinical stage of AD.[ Studies suggest that these individuals tend to progress to AD at a rate of approximately 10% to 15% per year.[ MCI patients also exhibited symptoms of language impairment.[ Some studies have held that confrontation naming and semantic verbal fluency tasks are effective in discriminating those with aMCI from the normal elderly,[ while others have argued that those tasks are not suitable for discriminating such groups.[ In a picture description task, aMCI exhibited low discourse abilities, including shorter discourse and more difficulties in maintaining a discourse topic, than those of normal elderly people, and had difficulty in understanding comprehensive discourse.[

To summarize the findings from the relevant studies above, not only one's memory and common cognitive abilities but also one's ability to perform discourse declines from the MCI stage. Studies have identified the importance of research into discourse in the aMCI stage as a prodromal stage of AD[ on the assumption that there is a close relationship between typical symptoms of dementia and discourse abilities.[ In addition, since it is impossible to explain a reduced cohesion of discourse only by considering one's difficulty in the lexical access at the word level, it has limitations to evaluate the language abilities of patients utilizing MCI with naming tasks only.

Concerning discourse analysis, existing studies have analyzed quantitative measures, such as content units, speech efficiency, the number of words, and anaphora, in order to assess discourse impairments. Some researchers have computed particular discourse behaviors that normally had not been considered in discourse analysis so as to compare and analyze ratios of modalizing discourse to referential discourse (M/R ratio).[ In spite of these various approaches for analyzing discourse by normal people or patients with neurologic communication disorders, limitations have arisen in applying discourse analysis in clinical settings because discourse analysis tends to be time-consuming. More specifically, analysis of discourse generally involves the recording of all utterances (about 10–15 minutes) followed by transcription. These complex and lengthy assessments are typically not feasible in clinical settings and particularly within the context of a hospital cognitive-communication screening. Furthermore, it is not simple to compare quantitative measures of discourse due to differences in the types of conducted discourse tasks, linguistic units (i.e., syllable, word, utterance) and time units (i.e., seconds, minutes) used in the analysis, and different languages.

In terms of diagnosis and intervention for diseases, early detection may alter the disease course. The main purpose of the present study is to explore whether subjective ratings of discourse (SR-D) may predict the quantitative measures of discourse tasks. In addition, we investigate the effectiveness of SR-D to identify and distinguish communicative decline in patients with aMCI.

Methods

Participants

Eighty community-dwelling elderly adults (40 aMCI patients and 40 normal elderly) residing in South Korea were recruited from May to December 2016. All participants were subjected to pre-evaluation questions in order to screen their hearing and vision. Participants had no history of neurological or psychiatric disorders. All participants gave informed written consent prior to participating in the study. This research complied with all applicable ethical rules and regulations and was approved by the Institutional Review Board of Daegu University (IRB#: 1040621-201405-HR-004-02).

Procedure and scoring

All participants were administered the following tests in order to more fully assess the characteristics of the population sample:

Mini-Mental Status Examination,[

verbal learning test (in Seoul Neuropsychological Screening Battery, SNSB),[ This measure involves studying a list of 12 words which is repeated 3-times. The total number of words recalled after each trial and the number of words recalled following a 20-minute delay.

Rey Complex Figure Test-copy (in SNSB),[

Short-form Geriatric Depression Test (SGDS),[ and

Korean Instrumental Activities Daily Living (K-IADL).[

We followed the criteria of the modified Petersen criteria for the diagnosis of aMCI[ and considered the diagnosis of aMCI if memory loss is the predominant symptom (below −1.0 SD on the verbal learning test). Participants whose MMSE and Rey Complex Figure Test (copy) scores deviated at least 1 SDs from this estimate were excluded from the study.[ Subjects showed depression were excluded from the study based on the SGDS score. Subjects with history of head trauma, epilepsy, stroke, or diagnosis of acute psychiatric disorder were also excluded from the study. There were no significant differences between the normal elderly and aMCI with regard to age (P =.06), level of education (P =.17), MMSE score (P =.31), RCFT (copy) (P =.56), SGDS (P =.10) and K-IADL score (P =.48), except in VLT–IR (P <.001) and VLT–DR (P <.01) (see Table 1).

Table 1

Subject characteristics.

With respect to the discourse, each participant was required to produce a spoken discourse sample. There are many types of discourse tasks (i.e., conversational, narrative, procedural). In clinical or research settings, picture description is widely used for eliciting narrative discourse samples from adults. Picture description has the advantages of providing a standardized approach to language sampling[ and allowing for performance comparison within and across groups.[ Stimuli included single pictures so that picture description tasks pose fewer demands on memory. However, in contrast to conversation and procedure discourse, the picture description is not a naturalistic task for participants. All discourse tasks do not elicit comparable levels or types of performance.[ Since diverse tasks for discourse complexity might detect subtle changes in language ability,[ we used 3 different types of tasks regarding discourse. The testing order was counterbalanced to reduce order effects.

Conversation

In order to assess the ability of conversational discourse, stimuli of a sub-test contained in “Protocole Montreal d’evaluation de la communication” (MEC)[ which developed to test for patients with traumatic brain injury was used. In the original version of the conversational discourse sub-test in MEC,[ the test procedure involved a ten-minute spontaneous conversation with the examiner on 2 different topics. In accordance with the original version, our examiner initiated 2 different topics of conversation of interest to the subject such as family, hobbies, or current events.

Picture narratives

Spontaneous speech samples in response to the “Beach” picture of the Western Aphasia Battery-revised version (WAB-R),[ were elicited in verbal modality. Subjects were instructed to tell the examiner about everything happening in the picture. Verbal responses were tape-recorded and transcribed.

Procedural discourse

Subjects were instructed to tell the examiner “how to make a noodle”. In order to elicit more verbal output in subjects whose narratives were insufficient or incomplete, prompts that did not reveal the content of the narratives such as “do you have anything else to add?” were offered. Discourse samples were audio-taped and transcribed verbatim.

Discourse rating

Subjective rating

Regarding the subjective rating, rating items for the conversational discourse in MEC[ were used. Among the 17 rating items in the original version of MEC, items related to cognitive dysfunction and language problem in aMCI were selected for use in this study. According to content validity, the 4th item (production of inappropriate or unexpected comments), 10th item (misunderstands what is said), 11th item (misunderstands indirect language), 12th item (remains indifferent to humorous comments), 14th item (use of monotone voice), 16th item (lack of facial expression), and the 17th item (inconsistent or absent visual contact) were excluded and the other 10 items were included in SR-D. Following the responses on the conversational discourse task, 2 speech pathologists rated the performance of participants utilizing a 3-point scale, with 0 indicating many; 1, often; and 2, none (Table 2).

Table 2

Selected 10 items of SR-D.

Objective analysis

Concerning objective analysis, the speech samples for the 3 discourse tasks were recorded and transcribed. The transcriptions were systematically checked by a second examiner. Data were analyzed for complexity, speech rate, and quality. Complexity was determined by the mean number of words per T-unit (MLTW) and TTR. Speech rate was determined by the mean number of syllables per second. Quality was determined by the CIU.[ CIU (%) was obtained by totaling the number of “relevant, truthful, non-redundant” facts[ for 3 discourse tasks.

Reliability

As a quality control measure in our study, 2 speech and language pathologists trained for 10 different communication behaviors checklist. They were blind to the participants’ clinical status and reevaluated 10% of the patients’ samples subjectively and objectively. The rates of agreement between the evaluators were 91.1% and 97.5%, respectively. A consensus was used in cases of discrepancy.

Statistical analyses

To identify the objective and subjective differences in performance for the 3 discourse tasks between 2 groups, the level of education and age were controlled through analysis of covariance (ANCOVA). With respect to the relationship between the SR-D and quantitative measures of narrative discourse, and the MMSE score was assessed utilizing multiple linear regression analysis, controlling for age, education, gender, and depressive symptoms. Partial correlation analysis was conducted to define the sub-items of SR-D which correlated with the objective measures. Following the result of the partial correlation analysis, ROC analysis was conducted in order to provide the cut-off score, sensitivity, and specificity of the selected sub-items of SR-D. Finally, the relationship between the selected 4 items of SR-D and quantitative measures of narrative discourse and MMSE score was assessed employing multiple linear regression analysis, controlling for age, education, gender, and depressive symptoms.

Results

Of the 80 participants, 4 aMCI subjects (5%) were found to exhibit depressive symptoms. The demographic information is shown in Table 2. An ANCOVA was conducted to confirm the differences in the objective and subjective measures between the normal elderly (NE) and aMCI groups after adjusting for age and education. In comparing the mean number of the objective and subjective measures, there were significant differences found in MLTW (F = 8.245, P <.01), CIU (F = 27.202, P <.001), and SR-D score (F = 192.114, P <.001) (Table 3). The multiple regression model with variables such as the MMSE score and objective measures accounted for 73.3% of the variance in the SR-D (F = 20.131, P <.001). Results indicated that only the CIU (B = .075, P ≤.001) and the MMSE (B = .339, P ≤.001) were significant predictors (Table 4).

Table 3

Comparison of narrative discourse between 2 groups.

Table 4

Relationship between MMSE score and quantitative measures of narrative discourse and SR-D.

To evaluate the relationship between MLTW, CIU, and sub-items of SR-D, partial correlation analysis was conducted after adjusting for age, education, and MMSE score. Among the 10 items of the SR-D, items 3 (“imprecise expression of ideas”), 6 (“lack of verbal initiative”), 7 (“excessive talking”), 13 (“loses track of conversation”) were significantly correlated with the performance of MLTW and CIU. Based on the results of the ROC analysis regarding these 4 items (short-form SR-D), the cut-off score was determined to be 7 points. Sensitivity and specificity were 100% and 75%, respectively (Fig. 1). The multiple regression model with the variables such as MMSE score and objective measures significantly predicted 57.8% of the variance in short-form SR-D (F = 9.791, P <.001). Results indicated that only the CIU (B = −.018, P <.05) and the MMSE (B = −.092, P <.05) were significant predictors (Table 5).

Figure 1

Result of analysis on ROC curve. cut-off score: 7 points (sensitivity: 100%, specificity: 75%, under the area:.983). ROC = receive operating characteristics.

Table 5

Relationship between MMSE score and quantitative measures of narrative discourse and short-form SR-D.

Discussion

Under the premise that there are some limitations in determining MCI in elderly individuals through language batteries mainly comprised of naming task, this study intended to examine the correlation between objective measurement and subjective measurement of discourse using a discourse task that reflects comprehensive and complex integrated communication skills, and to examine the clinical utility and significance of subjective measures of discourse.

Results showed that the performance on objective measures, the MLTw and CIU parameters, declined in the aMCI group compared to the NE group. First, MLTw refers to the MLTW, and thus, the decline of MLTw in the aMCI group reflects a small number of words in a sentence and shorter utterance length. As a consequence, there is the possibility that MLTw includes simple sentence structures. CIU is defined as “words that are clear in context and provide suitable information on the topic or task”.[ Therefore, lower CIU reflects insufficient contextual information. The lack of speech rate difference between the 2 groups, despite the decline in MLTw and CIU in the aMCI group, shows that although the 2 groups produce words at a similar speed, the aMCI group used relatively simpler sentence structure and insufficient contents, and produced utterance with declined efficiency and information communication in discourse. Further, a lack of statistically significant differences between aMCI and NE groups regarding TTR could be interpreted as that although the aMCI group produces various types of words during discourse, these words may not contain accurate information. To summarize these results in an objective perspective, although there is no change or increase in the physical amount of discourse produced by the elderly, the amount of contextually relevant information and the amount of communicated topic declined, which is consistent with previous studies.[ Therefore, these results suggest the importance of observing the differences in quality, as well as physical difference, in analyzing discourse.

The second notable finding is that the performance in the subjective measures of SR-D total score in the 2 groups differed. This result shows that subjective measures are sufficient to verify the decline in discourse ability in the aMCI group. To identify the effects of SR-D score and other variables in more detail, multiple regression analysis was conducted, in which MMSE, as well as CIU, could predict SR-D score by approximately 73%. Since CIU is a type of linguistic parameter, it is not surprising that the SR-D score can be predicted. However, MMSE is a screening test to measure overall cognitive function that includes questions assessing language function, as well as sub-areas of cognition such as attention, memory, and orientation. Therefore, the fact that MMSE could predict SR-D total score shows that discourse and various cognitive processes are related. This is in agreement with previous studies that reported that only the activation of the language region in the left hemisphere is required to perform language tasks at the single word or sentence level, while the right hemisphere and frontal cortex activation is required to accompany left hemisphere activation in order to produce discourse requiring not only basic language processing capacity such as semantic, syntax, and phonology but also higher cognitive processing capacity, including memory and executive function.[

However, SR-D total score reflects the sum of the 10 questions that assess various aspects of discourse, and thus a selective process was required to identify which question is more closely related to objective measures. Therefore, SR-D items were tested using partial correlation analysis in relation to CIU and MLTw, which showed significant results in objective measures, in which 4 items were found to have high correlation. First, “item 3: imprecise expression of ideas” and “item 6: lack of verbal initiative” could be interpreted as a measure of decline in the ability to access the lexicon and to retrieve lexical entry (i.e., words) to effectively express appropriate information in the aMCI group, instead of the deficit in the lexical-semantic storage itself. Therefore, these items reflect delay in retrieving words in a discourse situation, resulting in difficulty in starting utterance and decline in suitability of content. Further, “item 13: loses track of conversation” could be interpreted as a measure of decline in understanding the context or in executive function to organize and to summarize information in the discourse to understand the overall topic of discourse, due to the decline in attention or working memory. “Item 7: excessive talking” is related to the lack of effective communication despite the large number of utterance, due to lengthy expression to compensate for the defect in retrieving suitable words, and eventually leading to production of excessive utterances compared with the appropriate number of utterances that should be produced. Based on the results, the short-form SR-D presented in this study compressed into 4 questions is comprised of questions that can evenly evaluate the expression, reception and pragmatics domains of discourse. It can be concluded to show a correlation with efficiency of information delivery and various cognitive processes.

The third important result is that clinical effectiveness of the subjective measure, which could detect decline in language capacity, was verified through sensitivity and specificity analysis of the SR-D item. Traditionally, objective measures were effective in detecting defects, but its limitations included time-consuming transcription and analysis of discourse. In terms of economic opportunity costs, objective measures cannot be evaluated to be practical, since it is used in research rather than clinical diagnosis in general. Therefore, the evaluation of discourse utilizing a few items proven in its sensitivity and specificity could allow a wide use of such measures in not only research but also in clinical diagnosis.

Conclusion

Since lexical-semantic processing is not affected in every case of Alzheimer's disease,[ the discourse task that observes comprehensive communication skills can be utilized further to verify reduction of communication ability due to decline of cognitive ability. This study examined utility of subjective measurement of discourse. Indeed, subjective measures have limitations in producing different results dependent on the examiner's ability. However, the specialization of the scoring system as seen in the simplified 3-score measure (e.g., 0-1-2) in the SR-D scoring system could address such limitations, and online or offline training courses for examiners would promote the standardization of scoring for examiners and establish the reliability of results. This study can be utilized in examining decline of cognitive ability through quick and easy discourse screening test in patients with cognitive-communication disorders and briefly verify the declined dimensions of discourse, allowing for a more detailed and comprehensive picture on the neuropsychological profile of aMCI regarded as the prodromal stage of Alzheimer's disease. Further studies will be needed to confirm the external and internal validation for the new evaluation tool with conservative language tests. This study has some limitations in that we classified the participants only according to the amnestic type which has memory deficits. If a sufficient number of subjects are recruited in future studies, we will be able to identify the characteristics of the subtypes of MCI (amnestic type or non-amnestic type). In addition, we did not classify the 2 subtypes of aMCI (aMCI with single domain or multiple domains) in this study. Distinguishing between these 2 subtypes will necessitate checking visuospatial function, language ability, and frontal-executive function, in addition to memory function. However, due to the time constraints of the experiment, other tests besides VLT and RCFT could not be performed. Further studies will be necessary to check the characteristics of the sub-types of aMCI by adding additional tests such as fontal-executive function tests.

Author contributions

Conceptualization: JungWan Kim, Jihye Shim, Ji Hye Yoon.

Data curation: Jihye Shim.

Formal analysis: JungWan Kim, Jihye Shim.

Methodology: JungWan Kim.

Supervision: JungWan Kim, Ji Hye Yoon.

Writing – original draft: JungWan Kim, Ji Hye Yoon.

Writing – review & editing: JungWan Kim, Ji Hye Yoon.

Ji Hye Yoon orcid: 0000-0003-1403-2276.