A Systematic Review on methods of evaluate sentence production deficits in agrammatic aphasia patients: Validity and Reliability issues.

BACKGROUND: The grammar assessment in aphasia has been done by few standard tests, but today these tests cannot precise evaluate the sentence production in agrammatic patients. In this study, we review structures and contents of tests or tasks designed to find more frequent methods for sentence production ability in aphasia patients.
MATERIALS AND METHODS: We searched the Cochrane library, Medline by PubMed, Science Direct, Scopus, and Google Scholar from 1980 to October 1, 2013 and evaluated all of exist tests or tasks included in the articles and systematic reviews. The sentence production has been studied in three methods. It contains the use of sentence production in spontaneous speech, tasks designed and both methods. The quality of studies was assessed using Critical Appraisal Skills Program.
RESULTS: The 160 articles were reviewed and 38 articles were studied according to inclusion and exclusion criteria. They were classified into three categories based on assessment methods of sentence production. In 39.5% studies, researchers have used tasks designed, 7.9% articles have applied spontaneous speech and 52.6% articles have used both methods for evaluation production. Inter-rater reliability was between 90% and 100% and intra-rater reliability was between 96% and 98% in studied.
CONCLUSION: Agrammatic aphasia has syntax disorders, especially in sentence production. Most researchers and clinicians used both methods for evaluation production.

INTRODUCTION

Aphasia is an acquired communication disorder caused by brain damage and it result in impairments in speaking, listening, reading, and writing.[1] According to the National Institute for Deafness and Other Communication Disorders (1999), about 1 million people in the United States currently have aphasia.[1]

In aphasiology, “grammar” means the sentence structure, such as verb inflection as well as the relationship between sentence constituents. Agrammatism refers to pattern of sentence production that reflects an absence of grammatical structure.[2] Individuals with agrammatism show difficulty producing closed-class words that is, free-standing grammatical morphemes such as preposition, articles, pronouns, and auxiliary verbs.[2] Traditionally, it was argued that Broca's aphasia has a grammatical deficit and Wernicke's aphasia has a lexical semantic deficit. Furthermore, speakers with Broca's aphasia have more severe deficit in verb retrieval.[3] Because various kinds of sentence production may be involved in these patients, an in-depth assessment of the impairment is imperative.[2]

For the assessment of agrammatism in aphasia is to administer a standardized aphasia test battery such as the Western Aphasia Battery (Kertesz, 1982) or the Boston Diagnostic Aphasia Examination (Goodglass and Kaplan, 1983).[2] Standard tests to grammar assessment have three areas include testing sentence comprehension, testing verbs and verb arguments, testing sentence production and examining spontaneous discourse. Sentence comprehension tests are revised token test (McNeil and Prescot, 1978), auditory comprehension test for sentences (Shewan, 1981). However, they do not address all types of sentence-level problems. The most aspect of testing for agrammatic aphasia is the examination of verbs and verb arguments structures. Few published test is presently available for doing this, although several research laboratories around the world have developed protocols for this purpose.[2] One measure that has been used clinically for testing sentence production is the story completion test (Gleason et al., 1975; Goodglass et al., 1972). The examiner presents the model of the target sentence using the semantically reversed picture and then asking the patient to produce a similar sentence the target picture.[2]

Agrammatic output is commonly sampled in the clinic by asking an individual to produce a series of researcher made sentences that describe a nonstandard picture or sequence of pictures. Other regularly employed methods of data collection include a speaker re-telling a well-known story or engaging in dialogue on a topic suggested by the clinician. In contrast, the sampling of spontaneous speech produced for real-life communication with key conversational partners is still relatively rare.[4] In general, the tests can assess various aspects of grammar that this is not possible in assessing spontaneous speech and pictures description. Also, speakers with aphasia produce shorter syntactic units than the speakers without aphasia.[5] Therefore, clinicians cannot assessment all of grammar or syntax aspects perfectly. Mentioned standard tests above cannot evaluate sentence production detect therefore researchers begin to make individually tests or tasks to evaluate sentence production for agrammatic aphasia patients.

It seems a combination of all of exist tests for assessment data combined with quantitative and qualitative evaluating is necessary to provide a relatively complete process to evaluation[5] and assessment of treatment progress of sentence production in aphasia. A variety of different languages tests or tasks have studied of syntax or morphology in the world that are designed in sentence production. Authors did not find systematic review in diagnostic or assessment tests to sentence production assessment. In this paper in order to help speech-language pathologists clinicians and researchers, all of exist syntax tests or tasks in this patients based on their structure and methods are studied, classified and discussed their assessing areas The main goal of this study is to determine more frequent methods in sentence production assessment in agrammatic Broca aphasia.

MATERIALS AND METHODS

Literature search

We searched the Cochrane library, Medline by PubMed, Science Direct, Scopus and Google Scholar from 1980 to October 1, 2013. All studies were searched with this keyword: “(Sentence or syntax or syntactic) and production test (or assessment or evaluation or examination) and agrammatic aphasia” in these sites.

Study selection and eligibility criteria

In this study, all articles were studied included sentence assessment tests or tasks in adult agrammatic aphasia.

All of the syntax or sentence production articles or ways to determine sentence production assessment, all systematic review about them and all articles that were targeted therapy; but, it was used for the evaluation of sentence production were collected.

Eligibility criteria were:

The studies included test or task to diagnosis and assessment of sentence production abilities,

The test or task performed on agrammatic aphasia patients,

The test or task to determine of the methods to evaluate sentence production,

The articles were designed to evaluate sentence production ability before and after treatment (speech therapy).

Exclusion criteria were if:

The paper was reported a case study,

The language of test or task was not English,

The test or task did not assess sentence production specifically (such as they were evaluated verb production by verb naming or complete sentences, verb agreement or tense and they did not assess making sentences),

The paper was review article of previous study, and

The article was assessed bilingual aphasia patients.

Data extraction and abstraction

In this paper, the content of each test or task was evaluated and classified in separate tables. Then they were reported based on:

General characteristics of study (first author's name, publication year or study year, study design, and sampling method) and then were discussed on:

Characteristics of the study population (age and sex of studied participants and sample size, follow-up), and were completed

Type and duration of the interventional study, and finally were reported

Main finding of studies (validity and reliability of tasks or tests).

RESULTS

The data collected from the databanks showed that many studies have investigated the production of sentence.

A total of 160 articles were found in the sites that have studied sentence production. In this review, 122 articles were removed, and 38 articles were evaluated in detail.

In this section, articles were designed on three methods of sentence production assessment regardless of the purpose of studies. The results are presented in separate tables. Table 1 summarizes the features of tasks and tests designed for sentence production in agrammatic aphasia.[67891011121314151617181920] Table 2 includes the characteristics of the tasks designed for evaluate of spontaneous speech.[212223] Table 3 contains the tasks to evaluate of both sentence production and spontaneous speech.[2425262728293031323334353637383940414243] All studies were of two types of assessment and intervention, and it was the kind of speech therapy intervention. This therapy is a type of noninvasive interventions.

Table 1

The characteristics of tasks and tests designed for sentence production in agrammatic aphasia

Table 2

The characteristics of tasks designed for evaluate of spontaneous speech in agrammatic aphasia

Table 3

The characteristics of tasks and tests designed for sentence production and for evaluate of spontaneous speech in agrammatic aphasia

In 15 studies, researchers have used tasks designed to evaluate the sentence production. All studies have used the picture description for this purpose, but one study also has repetition task. Some articles have studied canonical (active) sentence only, but 9 articles (60%) has evaluated canonical and noncanonical sentences, 2 articles has evaluated unaccusative and unergative verb and 1 article has assessed the psych verb production. Duration of the test was not reported in articles.

Finding shows the 3 studies were used the spontaneous speech for the assessment of sentence production. In these articles was evaluated narrative speech by the story telling of Cinderella. Duration of the story telling was not reported in articles, and 1 article has studied test-retest reliability.

20 articles have used both sentence production tasks and spontaneous speech for sentence production evaluation in agrammatic aphasia patients. Three studies have assessed free conversation between patient and examiner, but 12 studies (60%) have evaluated story re-telling. Furthermore, 2 articles have applied the story telling, and free conversation and two of them have applied picture description in aphasia tests. The content of 4 tasks in these articles was about verbs and its assessment and 16 tasks have evaluated noncanonical sentences in agrammatic patients. Duration of storytelling and tasks was not reported. The validity of tasks or tests was calculated in 10 articles by assessing normal adults and the others were not reported. Ten articles were calculated the inter-rater reliability that the most of them were evaluated sentence production before and after treatment.

Figure 1 reports studies that have been used tasks or spontaneous speech or both methods for sentence production evaluation. It shows that most studies have used tasks in 2004-2008, but in the years 1999-2003 and 2009-2013 using both methods were common.

Figure 1

Number of articles in three methods for sentence production assessment in 1980–2013 (1 = 1980, 2 = 1984–1988, 3 = 1989–1993, 4 = 1994–1998, 5 = 1999–2003, 6 = 2004–2008, 7 = 2009–2013)

The result of evaluate the quality of articles using the Critical Appraisal Skills Program (CASP) for diagnostic test study showed based on data presented in articles could not response to the second question in CASP. So in all of articles response of the second question was “No.” It was impossible to continue to evaluate based on CASP guidelines.

DISCUSSION

The tests or tasks sentence production are useful tools for evaluating of speech output in aphasia patients. These tools can help clinicians in early diagnosis problems so it will not lose golden time of treatment. The correct selection of tasks can be very important based on the content and its quality, testing time and the scoring methods. In this study, we review all tasks and tests based on scientific methods of sentence production in the various language constructs.

Findings of this review suggest that there are different ways to evaluate the sentence production in agrammatic patients. It seems that before 1980, no study to use the scientific and systematic method for this purpose. From 1980, the sentence production is assessed by the different tasks or collecting spontaneous speech in story telling or free conversation. The researchers based their study's aim are used each of these two methods or both them. In this study, we found articles that have used specific tasks to evaluate better and more accurate sentence deficits in agrammatic aphasia. And almost there are 2 articles every year in this field. Tests or tasks design process has been going on continuously, and researchers in recent years have suggested the more accurate methods for the analysis of samples from patients. As mentioned above, the most of the research were designed and used of tests or tasks, but they have not done anything to standardize the tests or tasks. Some authors in later years, her tasks or tests have performed on the greater number of patients; they have solved their problems and have used other tasks (other lexical and syntactic skills) then have developed the standard test. It should be noted that very few standardized tests have designed and have published to evaluate the types of sentence production in this patients. In most studies was applied researcher made test. In this review, there were a standardized test designed to assess the comprehension and perception of various types of sentences and during the years 1986-2011 were used in different studies. Finally, it was published in 2011, and its psychometric properties have investigated.

The researcher has used 2 methods for sentence production evaluation in articles: “Tests or tasks” and “spontaneous speech.” The finding show that 39.5% of articles have used the method of design tests or tasks, the 7.9% have used spontaneous speech, and half of articles (52.6%) has applied both methods. Most studies have used both methods in the years 2009-2013. It can be inferred that the use of it can better demonstrate production deficits.

The 23 articles have examined the spontaneous speech totally, and speech sample have collected by the conversation between the patient and the examiner or the patient's family (34. 8%) or by re-telling stories (65.2% of studies). It seems the stories telling can be extracting more information of lexical, structural (syntactic) and morphological characteristics compared with free conversation in these patients. Because the patient uses stereotype and short sentences to communicate and cannot be determined what kind of grammatical elements has been removed in the speech? But in telling the story, the patients to describe the events happened and have to use grammatical elements such as a subject, objects, adjectives, adverbs, pronouns, and verbs in their sentences. Thus, examiner can be deleted or identified nongrammatical elements in the patient's speech and to judge it properly.

Review of articles show the researcher has used a variety of tasks or tests to evaluate of sentence production. These tasks include sentence completion, sentence repetition, making the sentence with the words given and sentence production priming. The few studies have used the combination of tasks in order to accurately speech analyze. By reviewing all studies (35 articles has used tasks or tests) it was found that the sentence production priming tasks was applied in 45.7% of studies (16 articles), making sentence was used 34.3% (12 studies) and sentence completion in the 11.3% of them (4 articles). In recent years, researchers tend to use of sentence production priming task. In this task, the examiner show one picture to the patient and produce a grammatical sentence after determining the role of elements in a sentence. The examiner asks the patient to perform the same process and produce a correct sentence. In this task is an emphasis on the self-production in patient. This task can be a good way to show one's ability or inability on sentence production.

The Broca aphasia patients have the brain injury in the frontal lobe (in 44 and 45 Broadman area), and they have the most frequent syntactic errors that are processed in these areas. Therefore, processing and production of noncanonical sentences than canonical sentences is difficult for them because the noncanonical sentences are complex. These sentences contain two types of syntactic movement: NP-movement and wh-movement. The wh-movement sentences is more complex the NP-movement sentences because in this type of sentence is necessary the grammatical elements to move to a higher nodes in the syntactic tree. The articles are considered complex sentences evaluating in these patients and in their tasks have assessed these sentences. The review of studies shows that two types of syntax movements have been studied in articles from 1986. The 25% of the article have studied the NP-movement and NP-movement sentences production. According to the main study, 28% of them have evaluated the type of NP-movement sentences, 48% have assessed the production of wh-movement sentences and 12% of articles have evaluated both types of syntactic movement. Furthermore, it is noted that 12% of them have assessed the scrambling. It is should be noted that in all studies was used the white and black line pictures for assessing of the sentence production.

Since the main objective of articles was to differentiate between normal subjects and patients; therefore, the finding showed the differential validity was calculated by the different statistical analysis. Half of the articles (55.3% of studies) have used the differential validity, but the other studies were not examined the kinds of validity. The reason could be that the pictures and tasks are clear for researcher, or the validity was not reported in the paper. Since any change in tasks or items (depend on easy or difficult items or type of the requested task) has a direct impact on patient response and the result of the test will affect directly, so the examiner through the assessment of validity of items and pictures in tasks can be sure the selected items is correct for evaluating of sentence production precisely. Hence, researchers should be evaluating the kind of validity of tests or tasks to select the most appropriate items.

There are three types of reliability in studies. The first type is the test-retest reliability. It was performed to ensure repeatability and consistency of responses in each test and calculated standard error of the mean in the test. This reliability has two types of intra-and inter-reliability. Inter-reliability reveals the examiner effects on the test. This reliability has been done in treatment studies in order to avoid the researcher bias. The test-retest reliability was calculated in 1 article. The second type of reliability measures the internal consistency of test, and it show that the test has integrity. This type of reliability is not reported in any of the articles. The third type is the editor's reliability. It is performed in tests that report qualitative scores and tests are done based on examiner judgment. Since the correct answers are clear in all of the tasks and tests and identify the correct answer is not dependence on the examiner view, so this reliability is not necessary in any of the tests and tasks. Nevertheless, it seems to evaluate the reliability of the tasks or tests increases the value of them and researchers can use it more confidently in other studies.

The sentences production is investigated in English structure in most studies. Only few articles were studied on other languages such as German, Dutch, Turkish, Arabic, Hebrew, and Russian. Since every language have the different structure therefore it is necessary are designed tests or tasks based on the language structure. The cross-linguistic studies can be useful to researchers. Therefore, it is necessary to study other languages such as Asian and African languages. It is noted that it is impossible examine the quality of the test or tasks based on CASP criteria in none of the articles. Because the second condition for this diagnostic or assessment study (which is the basic condition) was not reported in any of the articles.

CONCLUSION

The speech production evaluation is necessary the early stages of the disorder. By this way, we can assess the patient's syntax and morphology ability and resolve their problems using appropriate methods of treatment in early intervention. Without careful assessment of sentence production problems and treatment program will be based on early symptoms and it possible delayed the recovery in patients. In general, it can be stated that the sentence production have been studied in different methods. It can be inferred the use of both method of sentence production- narrative speech (free speech conversation or the story telling) and tasks designed- can get more information about the used or omitted various grammatical elements in agrammatic patients. Expert researchers and clinicians can use both of these methods and collected the complete language samples and tasks designed to extract more accurate data. Furthermore, the tests or tasks shall be designed the way that clinicians can do and analyze it, and Instructions of tests or tasks explained well and unambiguous. Therefore, it is essential that tasks and tests be standardized.

AUTHOR'S CONTRIBUTION

AM contributed in the conception of the work, conducting the study, revising the draft, approval of the final version of the manuscript, and agreed for all aspects of the work. ShJ contributed in the conception of the work, drafting and revising the draft, approval of the final version of the manuscript, and agreed for all aspects of the work.