How well do doctors understand a scientific article in English when it is not their first language? A randomised controlled trial.

INTRODUCTION: English is the lingua franca of science. How well doctors understand English is therefore crucial for their understanding of scientific articles. However, only 5% of the world's population have English as their first language.
METHODS:
Objectives: To compare doctors' comprehension of a scientific article when read in their first language (Norwegian) versus their second language (English). Our hypothesis was that doctors reading the article in Norwegian would comprehend the content better than those reading it in English.
DESIGN: Parallel group randomised controlled trial. We randomised doctors to read the same clinical review article in either Norwegian or English, before completing a questionnaire about the content of the article.
SETTING: Conference in primary care medicine in Norway, 2018. PARTICIPANTS: 130 native Norwegian-speaking doctors, 71 women and 59 men. One participant withdrew before responding to the questionnaire and was excluded from the analyses.
INTERVENTIONS: Participants were randomly assigned to read a review article in either Norwegian (n=64) or English (n=66). Reading time was limited to 7 min followed by 7 min to answer a questionnaire. MAIN OUTCOME MEASURES: Total score on questions related to the article content (potential range -9 to 20).
RESULTS: Doctors who read the article in Norwegian had a mean total score of 10.40 (SD 3.96) compared with 9.08 (SD 3.47) among doctors who read the article in English, giving a mean difference of 1.32 (95% CI 0.03 to 2.62; p=0.046). Age was independently associated with total score, with decreased comprehension with increasing age.
CONCLUSION: The difference in comprehension between the group who read in Norwegian and the group who read in English was statistically significant but modest, suggesting that the language gap in academia is possible to overcome. © Author(s) (or their employer(s)) 2021. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

We applied a randomised control design.

The authors were blinded to group randomisation while analysing the data.

Participants were a presumably homogeneous population regarding language, level of education and socioeconomic class.

We studied Norwegian doctors, who might not be representative for doctors in all other non-anglophone communities.

The questionnaire was in Norwegian (first language) for both groups, which could have introduced a linguistic switch cost for the group that read the article in English (second language).

Introduction

English is considered the global lingua franca of scientific research and publication, but only about 5% of the world’s population has the privilege of having English as their native language.1 Equity in global access to research is an important goal. Open access publishing and enabling low-income and middle-income countries access to collections of biomedical and health literature through the Hinari programme are important steps in this direction.2 They do not, however, help readers overcome their language barrier.3 Non-native English-speaking scientists experience disadvantage as they read, do research, publish and attend conferences in a different language than the one that is closest to their culture, thoughts and feelings.4 5

To date, research on second language comprehension has primarily targeted children, adolescents and immigrants to English-speaking countries,6 while little is known about professionals. Our research group published a study in 2002, which concluded that Scandinavian doctors’ ability to retain information from a review article was better when they read the article in their mother tongue than in English.7 To the best of our knowledge, no similar study has been conducted since.

Norwegian is the main official language in Norway, and the spoken and written language of daily life, in doctor’s offices and hospitals. Norwegian is also the tuition language of all medical schools in Norway. Children in Norway learn English as a second language in school from age six, and Norway is ranked third in the world for non-native English proficiency.8

The aim of our study was to learn about comprehension of science when presented in first versus second language. Our objective was to compare doctors’ ability to answer questions correctly about the content in a scientific article after having read the article in either their first language (Norwegian) or their second language (English). Our hypothesis was that doctors reading the article in Norwegian would comprehend the content better than doctors reading the article in English.

Methods

Study design

We performed a parallel group randomised controlled trial among doctors who attended a conference in primary care medicine in Oslo on 22–26 October 2018. Participants were randomised to read the same review article on paper in either Norwegian or English.9

Setting and participants

Approximately 1200 doctors working within primary care or public health attended the conference. Participants were consecutively recruited in the conference exhibition area. They were informed that we wanted to test whether different presentations of a scientific article affected reading comprehension and the ability to retain information. Before finally agreeing to participate, attendees were given written information about the study (online supplemental appendix 1), including that only doctors with Norwegian as first language were eligible for participation. Participants were given a small token of appreciation (an umbrella, value <£10).

Randomisation

We randomised participants by letting them pick an envelope from a box. The envelope contained the article in either Norwegian or English. Each participant would open their envelope and start reading the article, as an assistant set a digital alarm clock at 7 min. After 7 min, or earlier if the participant had finished reading, the assistant collected the article, handed the questionnaire to the participant and reset the alarm.

Data and variables

The topic of the article was the use of medication in pregnancy,9 thought to be a relevant issue for general practitioners. The article had been accepted for publication in the Journal of the Norwegian Medical Association but was not yet published at the time of the study. The English version was provided by the professional translation agency that is used routinely by the Journal of the Norwegian Medical Association.10

The article was 2300 words long. Reading time and the questionnaire had been separately piloted. Median reading time among pilot readers was 7.49 min for those reading in Norwegian and 8.35 min for those reading in English. We set the reading time to 7 min as we figured that time pressure would highlight possible differences between the two groups, and because we wanted to reflect the time pressure often met in clinical practice. We made minor modifications in the questionnaire based on feedback from the piloting.

Both groups filled in the questionnaire in Norwegian, which covered four components: (1) consent to participate, (2) demographic information on the participant, (3) background knowledge on the topic of the article and (4) questions related to the content of the article (online supplemental appendix 2). Demographic information included gender and age group (≤34, 35–44, 45–54, ≥55 years). We tested background knowledge on medication in pregnancy with a single multiple choice question with several correct answers (potential range −5 to 5). Questions related to the article content included five multiple choice questions (final scores potentially ranging from −9 to 12) and three open questions (range 0 to 8), adding up to a total potential score range from −9 to 20. Two of the authors (MR and SLS) independently scored the answers to the open questions based on prespecified guidance and blinded for language of the study article. They agreed in 83%–94% of cases dependent on variable. In cases of disagreement, consensus was reached by discussion (MR, SLS and RØ).

Power analysis

With random assignment to groups, independent samples of equal size, an alpha level of 0.05 and power (1-beta) of 0.80, a sample of n=128 would be necessary to detect an effect size of d=0.5, which would correspond approximately to a mean difference of 2 assuming SD=4 (two-sided test).

Statistical analysis

Data were processed and primary analyses performed blinded for language of the study article.

The primary outcome of our study was the total score on questions related to the article content. Groups were compared by two sample t-tests. Additional exploratory analyses were performed by simple and multiple linear regression, with total score on questions related to the article content as dependent variable and the following as independent variables: language, gender, age and background knowledge score. We tested for interaction between language and the following variables, respectively: gender, age and background knowledge. Finally, we performed two multivariate analyses: one limited to independent variables that were statistically significant in univariate analyses and one including all independent variables. Statistical significance was defined by an alpha level of 0.05. All analyses were performed in IBM SPSS Statistics V.25.

Patient and public involvement

This study did not include or directly relate to patients and was therefore done without patient involvement. Our research subjects are doctors, and the study was designed by doctors and doctors were involved at all stages of the process. The findings will be disseminated to the research subjects and to the general public through The Journal of The Norwegian Medical Association.

Results

One hundred thirty participants were recruited, of whom 64 read the article in Norwegian and 66 in English. One participant in the first group withdrew before responding to the questionnaire and was excluded from the analyses. All remaining questionnaires were complete, and this resulted in a final sample of 129 (63 vs 66) (figure 1).

Figure 1

Flow of participants.

Table 1 presents demographic data and scores on background knowledge for each group. Participants who read the article in Norwegian had a mean total score on questions related to the article content of 10.40 (SD 3.96) compared with 9.08 (SD 3.47) among participants who read the article in English, giving a mean difference of 1.32 (95% CI 0.03 to 2.62; p=0.046).

Table 1

Characteristics of study population and scores on background knowledge

Variable	Norwegian text(n=63)	English text(n=66)
Gender
Female	35 (55.6)	35 (53.0)
Male	28 (44.4)	31 (47.0)
Age (years)
<34	17 (27.0)	22 (33.3)
35–44	17 (27.0)	16 (24.2)
45–54	12 (19.0)	10 (15.2)
>55	17 (27.0)	18 (27.3)
Score on background knowledge, mean (SD)*	1.33 (1.32)	1.06 (1.12)

Values are numbers (percentages), unless stated otherwise.

*Range –5 to 5.

The results from the linear regression analyses are shown in table 2. Participants >55 years had a mean total score of 8.29 (SD 2.87) compared with 10.41 (SD 4.35) among participants <34 years (unstandardised B −2.13 95% CI −3.81 to −0.44; p=0.014). The effects of language and age on total score were also statistically significant in both multivariate analyses (table 2). We found no statistically significant interactions (data not shown). The assumption of normally distributed observations was confirmed by visual inspection of histograms and QQ-plots.

Table 2

Linear regression with total score related to the article content as the dependent variable

Variables	Univariate analysis		Model 1*		Model 2†
	Unstandardised B (95% CI)	P value	Unstandardised B (95% CI)	P value	Unstandardised B (95% CI)	P value
Language
Norwegian	Reference		Reference		Reference
English	−1.32 (−2.62 to −0.03)	0.046	−1.36 (−2.62 to −0.11)	0.034	−1.29 (−2.55 to −0.03)	0.046
Gender
Female	Reference		–		Reference
Male	0.11 (−1.21 to 1.43)	0.871			0.83 (−0.51 to 2.16)	0.222
Age
<34	Reference		Reference		Reference
35–44	0.47 (−1.24 to 2.18)	0.588	0.36 (−1.33 to 2.05)	0.673	0.61 (−0.11 to 2.32)	0.486
45–54	−1.37 (−3.29 to 0.56)	0.163	−1.51 (−3.42 to 0.39)	0.118	−1.46 (−3.37 to 0.44)	0.131
>55	−2.13 (−3.81 to −0.44)	0.014	−2.19 (−3.85 to −0.53)	0.010	−2.21 (−3.91 to −0.51)	0.011
Background knowledge‡	0.40 (−0.13 to 0.94)	0.140	–	–	0.31 (−0.22 to 0.84)	0.243

Univariate and multivariate analyses (model 1 and model 2).

*Adjusted for variables statistically significant in univariate analyses.

†Adjusted for all independent variables.

‡Range −5 to 5.

Discussion

We investigated whether reading comprehension of a scientific article was best in the subjects’ first or second language by randomising 130 native Norwegian doctors to read the same article in either Norwegian or English, and then answer questions about the article content. Doctors who read the article in their first language had more correct answers than the doctors who read the article in English. The difference in score was small, but statistically significant.

Some of us published a similar study in 2002 among Norwegian, Swedish and Danish doctors. All three groups retained information from a given article better when read in their mother tongue versus English, with a median (IQR) of 4 (3–6) vs 3 (2–4), respectively (p=0.01).7 The two studies are not directly comparable, but our results indicate that the difference might have diminished over the past two decades. Our finding is also in line with previous research in the field of bilingualism.11

Comprehension is a complex process that is hard to define and even harder to test. It is the ability to process text, to decode its meaning and to integrate that with what the reader already knows about the subject. We tested comprehension in the same manner as at examinations in many medical schools, with a mix of multiple choice and open questions. We tested all participants in Norwegian, a choice we made based on the presumption that true comprehension should be more than simple recollection, that is, if you read in a second language, you should be able to answer questions about it in your first language. A possible pitfall with this design is that the participants asked to read the article in English in a Norwegian context and then answer questions in Norwegian are subject to what is known as linguistic switch costs.12 Switch costs refer to the cognitive burden of switching languages, which results in longer processing times or higher error rates. This could explain the difference in scores between the groups.

A strength of this study is that research subjects were recruited from a presumably homogeneous population regarding language, level of education and socioeconomic class, and then randomised. In the field of bilingualism, this is quite rare, as most studies on second language comprehension test bilinguals with a monolingual control group13; often the bilingual group consists of a minority population and the monolingual group consists of the cultural majority. This comes with a set of systematic differences between the groups regarding culture, education and socioeconomic class. Furthermore, testing bilinguals versus monolinguals is problematic in itself as it is well documented that bilingualism per se offers a cognitive advantage in some tasks related to executive function.14

One participant in the group who read in Norwegian withdrew before responding to the questionnaire and was excluded from the analyses. Depending on the reason for non-response, the estimated difference between groups might be slightly biased in this complete case analysis.

Our findings are probably not generalisable to all non-anglophone scientific communities for many reasons. For one, English and Norwegian are both Germanic languages which means they have more linguistic features in common than do, for instance, English and Russian or Hindi. Further, proficiency in English is high in Norway compared with most other countries.8 Norwegian doctors do also have better access to the Internet and to research articles both in English and in their first language than do many colleagues in low-income and middle-income countries. Hence, our results might be a best-case scenario for comprehension of science in a second language. Similar studies in other countries would yield additional insight.

Implications

In order to level the playing field in global academia, we must acknowledge that language is intrinsically linked to power and privilege.1 If the goal is to leave no-one behind,15 funding of education in English and academic English in low-income countries is essential. In this mindset, reaching those furthest behind would mean reaching the 6 billion people who do not speak English at all. English as a common language in science offers unprecedented possibilities for cooperation, mutual understanding and dissemination of research, and it can also be a democratising institution if extended to all.