Applying a Computer-Assisted Tool for Semantic Analysis of Writing: Uses for STEM and ELL.

In addition to human, close reading of student text with rubrics for assessment, educators use nonhuman, distant computer-assisted tools to help quantitatively measure otherwise qualitative keywords to prevent bias in grading and help read beyond the sentence for underlying cognitions. We apply the Linguistic Inquiry and Word Count (LIWC) software tool to analyze different forms of student writing used in STEM education and research to assess writing of native English speakers and non-native English Language Learners (ELLs), including international students. Available in several languages, LIWC measures four summary variables, Analytical Thinking, Clout, Authentic, and Emotional Tone, to provide outputs as raw word counts, as percentages of words used relative to the text compared with a dictionary of words in categories and sub-dictionaries, and as scores correlating these words algorithmically based on a dictionary of terms associated with underlying meanings. This tool can help measure student personal reflective writing for underlying psychosocial indicators or the cognitive and analytical process in other science writing. By selecting key variables, or creating a personal dictionary, LIWC can be used to analyze scientific writing to detect progressive development of student analytical writing from early draft to final version for different informal and formal writing styles. We share methods, examples, and the potential for using LIWC measures of cognitive processes for different measures of student writing in science courses.

INTRODUCTION

Educators use human, close reading of student-written text to judge content, grammar, and quality, with rubrics to guide assessment. The qualitative grading process searches for coded keywords, patterns, and emergent themes. Because educator training, experience, and personality influence evaluation, assessment carries limitations for deeper analysis of hidden learning and potential for implicit bias. If educators use non-human, distant reading with computer-assisted scoring, counting key words for analytic scores can provide a more quantitative output to assist in checking bias, e.g., automated tools check spelling, word counts, and percent similarity scores for plagiarism. There are other computer-assisted tools, such as the Linguistic Inquiry and Word Count (LIWC) software (1), that can be employed for semantic analysis to help measure the latent learning that takes place within the writing process, be it the psychosocial process in reflective writings or the cognitive and analytical process in other science writing.

A comprehensive report (2) on assessment and cognitions provides background on the sociocognitive framework connecting writing and thinking skills with writing assessment; it supports that writing in different genres helps learners explore ideas. The complexity of writing as a construct, use of dictionary codes with different genres, and reliability of scores as predictors are considerations for limitations of automated rating. Ongoing discussions raise challenges regarding technical inadequacies affecting accuracy, the lack of sensitivities that human raters provide, and the impacts of validity on educational consequences; however, if reliable, scores can be used in a rubric to promote learning and improve instruction (3, 4). To more deeply assess student writers by analyzing their writing, dictionary-based semantic analysis is used to detect latent cognitions, such as underlying analytical thinking, emotions, and other features applicable to learning—all well documented for LIWC (5–14).

The LIWC tool, free trial with limitations (http://www.liwc.net/tryonline.php) or inexpensive LIWC2015 software (http://liwc.wpengine.com/), has been iteratively developed and empirically tested by a team of computer programmers, linguistic specialists, and psychologists. It searches different styles: expressive, reflective, formal writing, and oral conversations transcribed into written text (5–7). Since language semantics encompass deeper meaning and content hidden within word vocabulary and context, the software uses a highly developed dictionary to analyze themes. Thus, we vetted LIWC to study different personal reflective writing and science communication exercises in an intensive-writing biochemistry laboratory course.

A meta-analysis of holistic and analytical rubrics (4) supporting the benefits of intentionally designed rubrics, scaffolded learning practices, and peer evaluation using rubrics prompted our studies of several non-native English-speaking international student cases using both qualitative and quantitative writing analysis (8). Our studies to detect underlying patterns of thinking were founded on previous uses of LIWC: students’ emotion-based and meaning-based coping in the ability to manage stress and wellbeing (9, 10); confidence, personal development, and social belonging (5, 11–12); and increased cognition due to course workload, active learning, and language switching for ELL and international students (9, 11).

Together with these prior examples, our work (8) more fully describes schema theory and latent semantic content analysis using automated coding, providing an example of biased grading ameliorated using LIWC to help read beyond the sentence for international Asian ELLs to reduce hand-graded bias even when grammar errors persist. Some LIWC measures provide psychosocial indicators; others we adapted, analyzed, and further applied for scientific writing (examples shown here, Appendix 1). It is our hope to develop a full heuristic model from detectable LIWC patterns to better understand the writers as well as their writing.

Here, we provide example analyses of institutional review board (IRB)-approved student samples to show applied use of LIWC in our writing-intensive science course with several genres, measuring a variety of factors that influence student learning, inclusion, success, and retention in STEM of native-English speakers and non-native English Language Learners (ELLs), including international students.

PROCEDURE

Preparing for software use

The LIWC2015 tool sequentially counts words in text files and compares them with built-in dictionaries or a custom dictionary. The language manual with additional empirical references (7) and the software operator’s manual for Mac or PC have dictionaries available in several languages for international studies (1). General steps outlined here are expanded, with samples and resources providing LIWC history, development, and tips for data organization and use (5, Appendix 1).

Obtain administrative IRB approval for all student writing for research purposes to guarantee de-identification and provide informed consent for students sharing their writing.

Decide the writing style for hypothesis testing or evaluation before choosing appropriate variables. The LIWC software categorizes raw word counts, and four summary language variables. Clout (confidence), Authentic (honesty vs. hedging), and Emotional Tone (affect) are scored as the percentage of words used in the text compared with a dictionary of words in categories and sub-dictionaries. Analytical Thinking is determined algorithmically using correlated words based on a dictionary of terms associated with underlying meanings (Table 1).

TABLE 1

Sample LIWC2015 code dictionaries.

Category	Abbrev	Examples	Words in category
Word count	WC
Summary language variables
Analytical thinking	Analytic
Clout	Clout
Authentic	Authentic
Emotional Tone	Tone
Words/sentence	WPS
Words>6 letters	Sixltr
Linguistic dimensions
Total function words	funct	it, to, no, very	491
Total pronouns	pronoun	I, them, itself	153
1st pers singular	I	I, me, mine	24
Articles	article	a, an, the	3
Other grammar
Common verbs	verb	eat, come, carry	1,000
Common adjectives	adj	free, happy, long	764
Comparisons	compare	greater, best, after	317
Interrogatives	interrog	how, when, what	48
Numbers	number	second, thousand	36
Quantifiers	quant	few, many, much	77
Psychological processes
Affective processes	affect	happy, cried	1,393
Anxiety	anx	worried, fearful	116
Cognitive processes	cogproc	cause, know, ought	797
Insight	insight	think, know	259
Causation	cause	because, effect	135
Discrepancy	discrep	should, would	83
Tentative	tentat	maybe, perhaps	178
Certainty	certain	always, never	113
Differentiation	differ	hasn’t, but, else	81

Adapted with permission from LIWC language manual (7).

This small sample of words in each category from a larger dictionary provides example words and the total number of words in each category. As educators develop their own rubric, such as using terms for Comparisons as previously shown (8), they can include Linguistic Inquiry and Word Count (LIWC) codes useful for analysis of a variety of text styles using total raw Word count and four Summary Language Variables: Analytical Thinking, Clout, Authentic, Emotional Tone. Categories such as Linguistic dimensions include subcategory function words, pronouns, and use of “I.” This category is used in expressive and reflective writing along with the category Psychological processes, which includes subcategories Affective processes and Anxiety, but is not used with scientific writing. The frequency of pronouns and grammar is used in scientific writing to detect formality. Other Grammar includes subcategories Comparisons, Quantifiers, and other variables (not shown) such as Punctuation. These, along with the category Cognitive processes, are useful for both reflective and scientific writing.

Applying theory

Evidence-based research and theory using LIWC to study different written text styles provides a foundation for adaptable applications. Developed to search for terms associated with latent traits of certain psychological conditions, this tool can detect genres of writing, hidden internal personality, and cognitions. It has been used in internally and externally validated studies to measure features which, when extracted by the program, act as tells for predictive outcomes (8, 12–14); however, caution is advised depending on the writing genre.

Choose LIWC variables and categories (7); examples are provided (Appendix 1). Prior to assessing student work or conducting research, form predictions and determine codes relevant for 1) psychosocial analysis useful for personal, reflective writing as opposed to 2) scientific writing analysis.

All four summary variables for psychosocial analysis are used for personal reflective, expressive writing, or graduate school personal statements measuring cognitive processes, confidence, and emotion. Some variables track student emotional health, e.g., the affect variable detects emotion from expressively written text following prompts. Pennebaker and others link health or confidence to pronouns as indicators— the more “I,” the more inward thinking, less esteem, and more potential for depression or other indicators of poor health (12–14).

Not all categories are appropriate with scientific writing, e.g., pronouns are not for emotion but only as indicators of formality, e.g., popular science news or more formal layperson writing permits more pronoun use than formal scientific writing. Different variables, e.g., use of quotes, can detect changes in writing style useful as predictive analytical metrics for formative or higher cut-off formality for summative grading assessing scientific writing. The LIWC tool detects progressive development of student analytical writing over time, from early draft through rewrites, tracking the reduction of contractions, quotes, or parentheses in a final improved version. Pre/post analysis quantitatively correlates Analytical Thinking and subcategories Cognitive, Quantifiers, and Comparisons scores examined by two independent raters (Fig. 1), correlated by two hand-graders using rubrics (8), and further examined by an external evaluator.

FIGURE 1

Screenshot of sample Linguistic Inquiry and Word Count (LIWC) output. LIWC analysis of student Biochemical (BQA) draft questions Q (reflective) and answers A (scientific) writing. Export to Excel shows left column filenames or de-identified student numbers from a file opened in LIWC software. Top row sample categories are selected based on the desired writing style analysis. Word count (WC), and summary variables Analytical Thinking, Clout, Authentic, and Emotional Tone are raw data or algorithmically determined and useful for many writing styles. Some variables help determine the complexity of writing: Words per sentence (WPS), Words > 6 letters (Sixltr). Categories are percentage scores of the number of words from the text relative to total word count, e.g., Pronouns, I, or other me personal pronouns are used for different purposes in different styles of writing, such as determining formality in scientific writing style. Categories are algorithmically nested under summary variables, e.g., Analytical Thinking summary variable comprises Comparison, Quantifier, Cognitive Processes, and others, according to the LIWC dictionary. Emotional Tone summary variable comprises Affect, Anxiety, Positive, Negative, and others. A variety of punctuation, e.g., Quotes, Apostrophes, Parentheses, etc., are useful for tracking scientific writing formality. We define formality of scientific writing per our grading rubric as having these features: zero to low personal pronouns, no quotes, no contractions, no apostrophes except the four expected for two 5′ and 3′ DNA primer ends in a Methods section. Scores matched hand-graded appropriate use of parentheses for defined abbreviations, citations, and chemical names but were not overused in layperson writing with increased definition of scientific terminology. “I” (0.00) was consistent with BQA scientific answers A, whereas questions Q had allowable pronouns in the reflective style. Higher use of quotes, apostrophes, and parentheses was detected with less formal writing such as use of contractions (“Conc’t sample”), which was found in other writing samples and corrected upon later rewrite of draft (BQA1D, BQA2D, BQA3D) to final versions (BQA1F, BQA2F, BQAS3F). LIWC scores were matched to hand-graded counts by two independent raters and reviewed by an external evaluator, with >95% agreement, and two additional independent in-class graders for comparison, quantitatively assessing levels of Comparisons, Quantifiers, and Cognitive reasoning and Analytical thought (8, Appendix 1 examples).

Prepare data by gathering a selection of electronically stored word or PDF documents. Assessors decide the desired level of data-cleaning, e.g., foreign language translation, error-prone or corrected spelling or grammar, pervasive mathematical and scientific jargon, etc. In studies to detect whether computer-assisted analysis could support ELL writing challenges, semantic writing analysis was useful despite grammar issues (8–9, 11).

With student identifiers removed, check all written documents for readability, group them in a labeled file, organize data; check for fidelity of text data with any errors or potential loss due to transcoding into word files, e.g., extra text from headers.

After purchasing the software passwords, open files in the LIWC2015 software and export output data to Excel (Fig. 1). Examine output as raw word counts, algorithmic scores, or percentage of words per total word count analyzed for summary variables, categories, or additional features by searching specific keywords in personalized dictionaries. Evaluate results by prioritizing datasets to incorporate dictionary codes most useful for the intended analysis into a rubric. Use graphical analysis to visually see patterns in student writing (Fig. 2) for ongoing studies and continued validation to support non-biased grading of students as individual learners and additional analyses (8, 12, 15–18).

FIGURE 2

Visual graphical sample for comparison. Once data are graphed, detected patterns are more easily seen between the Biochemical (BQA) Question Q part, which is reflective and poses a question with more personal pronouns “I”, and the Answer A part, which is formal scientific writing and has a higher Analytical Thinking score. Some progressive improvement can be detected from drafts (BQA1D, BQA2D, BQA3D) to their final versions (BQA1F, BQA2F, BQAS3F), with higher Analytic scores algorithmically detecting more formality in the scientifically written answer A section and with some detectable increases in total scores, e.g., BQA2 improved from 86 to 87, and BQA3 improved from 87 to 89. Clout as a measure of confidence is higher in some question sections than others, demonstrating that confidence can vary per different topics in the reflectively written question Q section. While Clout per percentage words in the dictionary as an indicator of confidence can be useful in psychosocial research studies, it is not useful in a written rubric for grading. Variables are not used for all genres, e.g., Clout psychosocial indicator is not used in the Answer portion, which is scientific and not reflective writing. These computer-generated scores were matched with hand-grading and visual inspection by the two authors. Examples are provided in Appendix 1.

CONCLUSIONS

As student writing is assessed with qualitative, close reading, supplemented by application of more quantitative, distant analytical tools like LIWC in science-writing courses, educators can detect different emergent patterns from a semantic analysis of mixed-text student samples. This analysis helps in the investigation of the psycho-social aspects of learning using specific LIWC variables and categories to detect underlying constructs in reflective versus scientific writing to capture predictive results. Because LIWC is useful with non-native English speakers and also comes in several languages, this tool can be applied internationally. Although some patterns appear predictive (8), once deemed reliable for grading purposes, LIWC codes can be further incorporated into a rubric to reduce bias in grading with objective numeric or algorithmically generated scores. This makes LIWC an interesting, adaptable tool with the potential for studying how learners learn through baseline, pre/post-studies, and identification of profiles in blinded studies for different measures of STEM and ELL student writing in science writing.

Click here for additional data file.