First quality score for referral letters in gastroenterology-a validation study.

OBJECTIVE: To create and validate an objective and reliable score to assess referral quality in gastroenterology.
DESIGN: An observational multicentre study. SETTING AND PARTICIPANTS: 25 gastroenterologists participated in selecting variables for a Thirty Point Score (TPS) for quality assessment of referrals to gastroenterology specialist healthcare for 9 common indications. From May to September 2014, 7 hospitals from the South-Eastern Norway Regional Health Authority participated in collecting and scoring 327 referrals to a gastroenterologist. MAIN OUTCOME MEASURE: Correlation between the TPS and a visual analogue scale (VAS) for referral quality.
RESULTS: The 327 referrals had an average TPS of 13.2 (range 1-25) and an average VAS of 4.7 (range 0.2-9.5). The reliability of the score was excellent, with an intra-rater intraclass correlation coefficient (ICC) of 0.87 and inter-rater ICC of 0.91. The overall correlation between the TPS and the VAS was moderate (r=0.42), and ranged from fair to substantial for the various indications. Mean agreement was good (ICC=0.47, 95% CI (0.34 to 0.57)), ranging from poor to good.
CONCLUSIONS: The TPS is reliable, objective and shows good agreement with the subjective VAS. The score may be a useful tool for assessing referral quality in gastroenterology, particularly important when evaluating the effect of interventions to improve referral quality. Published by the BMJ Publishing Group Limited. For permission to use (where not already granted under a licence) please go to http://www.bmj.com/company/products-services/rights-and-licensing/.

Multicentre design gives better external validity.

Wide variety of gastrointestinal indications encompassed by score.

Score does not assess appropriateness of the indication for the referral.

Potential interindividual differences in the use of the visual analogue scale.

Introduction

General practitioners (GPs) refer numerous patients to specialised healthcare services every year, and the referral rates are increasing worldwide.1 2 A recent study revealed a referral rate of 13.7% with great variations between GPs.3 The referral letter is a key document for the communication between the GPs and the hospital consultants,4 5 and its content and quality are essential for the scheduling and prioritisation of patients. Incomplete, erroneous or extensive referral letters may delay the admission of patients to secondary healthcare services, which may result in delayed diagnosis and a poorer prognosis. It is well documented that referral letters of low quality are prevalent.5–21

In Norway, referral letters are sent at the GP’s discretion and generated by using free text in a standard template created by the Norwegian health authorities22 (table 1). The template includes the urgency of the referral, but does not include symptom-specific or indication-specific criteria. It is very likely that specific symptoms and clinical findings are crucial for deciding the urgency of the referral.23 The GPs optionally add relevant information about supplementary workup, like laboratory tests and imaging results, by copying and pasting these results to the referral text. Subsequently, the referral is transferred electronically by a secure system to the hospital.24 At arrival, a consultant assesses the referral letter and prioritises the urgency of the referral. If the consultant considers the referral indication inappropriate, it may be rejected. However, the legislation prohibits rejection due to poor quality.

Table 1

General requirements for referral letters, according to the Norwegian national referral guidelines, including the frequency of reporting in this study

Referral information	N (%)
Administrative information (patient, GP, referral institution)	325 (99.4)
Referral diagnosis	319 (97.6)
Type of referral (examination, workup, advice)/expected action	313 (95.7)
Urgency of referral	58 (17.7)
Critical information	68 (20.8)
Allergies/infectious disease	66 (20.2)
Other ongoing critical diseases and treatments	6 (1.8)
Personal information	181 (55.4)
Family history of disease	94 (28.7)
Social history (work, school, family, living)	122 (37.3)
Alcohol and tobacco history	60 (18.3)
Previous medical history	259 (79.2)
History of the current disease	327 (100.0)
Findings	267 (81.7)
Clinical examination	164 (50.2)
Laboratory workup	173 (53.9)
Radiology/other	119 (36.4)
Current medicines	271 (82.9)
Patient informed of referral	0
Total referral letters	327

GP, general practitioner.

The Norwegian Prioritization Guideline for gastroenterology (NPGg),25 created by an expert group of gastroenterologists, states nine main indications for referral to gastroenterology services (open access endoscopy and consultations); dyspepsia, dysphagia, chronic abdominal pain, change of bowel habit, diarrhoea, constipation, gastrointestinal bleeding, weight loss and jaundice/elevated liver enzymes. The guidelines aim to cover 75–80% of gastroenterology referrals. Bowel cancer screening and polyp surveillance is not a part of the indications because this is taken care of either in the hospital system or by population-based trials for bowel cancer screening.

Despite the well-documented quality issues, no validated and objective tool to assess the quality of referral letters to the specialist healthcare services exists.

An objective, standardised score would probably be beneficial to ease the assessment of referral quality and to evaluate interventions intending to improve the quality of referral letters.

The aim of the study was to develop an objective, relevant and reliable quality score for the evaluation of referral letters for the most common indications for referral to gastroenterologists, and to validate the score in a clinical setting.

Materials and methods

Development of the score

We used the nine core indications for referral to gastroenterologists specified in the NPGg:25 dyspepsia, dysphagia, chronic abdominal pain, change of bowel habit, diarrhoea, constipation, gastrointestinal bleeding, weight loss and jaundice/elevated liver enzymes.

For each of these nine indications, we created a list of 29–36 relevant medical variables based on UpToDate,26 the Norwegian Electronic Medical Handbook for Doctors (NEL)27 and the NPGg.25 Administrative information including sex and date of birth of the patient is included in referral letters by default, as well as name, health personnel identification number and address of the referring physician, and therefore we did not include this information in the nine lists.

Between November 2013 and March 2014, we invited 39 gastroenterologists within the South-Eastern Norway Health Region to participate in a web-based study. They recorded demographic information such as sex, age, experience and workplace before they started the survey. For each of the nine indications, we asked the participating gastroenterologists to select the 15 most important variables for assessing and prioritising referrals. The importance of the variables was categorised from three points (most important) to one point (less important) with five variables in each category. The remaining variables were given zero points. We then summarised the points assigned to the individual variables for each indication from all the returned questionnaires, and selected the 15 variables with the highest sum of points to comprise the final variables in the score. The five highest rated variables were classified with three points, the next five with two points and the last five with one point.

After a period of a minimum 6 weeks, we repeated this process to check for reliability of the values assigned. Only the results from the first round were used in the final score. This resulted in a symptom-specific Thirty Point Score (TPS).

Validation of the score

Between May and September 2014, seven primary gastroenterology referral centres in South-Eastern Norway Health Region collected 327 referral letters, 21–50 for each of the nine indications stated in the NPGg.25 Patients were mostly referred for open access endoscopy as well as consultations.

One or two gastroenterologists in each participating centre collected and assessed consecutive referrals within the nine indications. They rated the quality of the referral letter on a 10 cm visual analogue scale (VAS), where 0 cm indicated the worst possible quality and 10 cm the best possible quality. We chose VAS as a comparator to the TPS to assess the external validity in the validation process, assuming that referral letters containing all essential information for assessment and prioritisation of the referred patients would also yield a high VAS.28 The gastroenterologists also recorded whether the referral letters contained too much or too little information, and whether they were unstructured or illegible. Patient age and gender were recorded before all patient data were removed from the referral letters and they were handed over to the study team.

Subsequently, one researcher from the study team assessed all the referral letters according to the TPS. Both the presence and absence of signs and symptoms were given equal points, as long as they were reported adequately in the referral letter. We also recorded demographic data about the referring physicians (sex, age, county, size of patient list, etc) from publicly available records, and whether the referral letter complied with the national guidelines for referral letters to the specialist healthcare services.22

Information that was not available to the consultant through the referral letter at the time of assessing the referral, for example, results from laboratory tests and radiology that were communicated later in the referral process, was not included when calculating the TPS.

To check for intra-rater and inter-rater reliability, we randomly selected 25% of the referral letters from each indication and reassessed them with the TPS. This was done after a minimum of 6 months by the same researcher from the research team and a second independent gastroenterologist. These ratings were done completely blinded from each other and from the results of the first rating.

Statistics

We present descriptive statistics as means with their 95% CI or as proportions. We use intraclass correlation coefficient (ICC) for continuous measures and weighted κ statistics for categorical measures to describe reliability of the gastroenterologists' selection of score variables. ICC for average measurements was used to calculate intra-rater and inter-rater reliability of the TPS assessment of referral letters. ICC values were interpreted as: >0.75=excellent, 0.40–0.75=fair to good and <0.40=poor.29 The κ values were interpreted as: >0.80=very good, 0.61–0.80=good, 0.41–0.60=moderate, 0.21–0.40=fair and <0.21=poor.30 Univariable and multivariable linear regression analysis was performed to determine whether patient-related or doctor-related factors were associated with changes in quality of the referral letter, using a manual backward elimination procedure. Any variable with a p<0.25 from the univariable analysis was considered a candidate for the multivariable model. We used the Pearson correlation coefficient to assess the correlation between the VAS and the TPS and ICC for average measurements to assess agreement between the two measurements. We assessed any differences in the VAS/TPS between the different centres by using one-way analysis of variance (ANOVA). The significance level was set at 0.05. All calculations were performed using the IBM SPSS V.21 (IBM SPSS, Chicago, Illinois, USA).

Power estimation

Sample size estimation was performed to investigate the association between the VAS and the TPS. The correlation coefficient (r) was anticipated to be 0.60. Considering a probability of a type I error of 5%, 80% power and a two-sided test, at least 19 referral letters were required for each indication. Thus, during the inclusion time, a minimum of 21 referral letters were collected for each indication. Since some indications were more common than others, the number of referral letters for each indication varies upwards.

Results

Selection of TPS variables

Of the 39 invited gastroenterologists, 32 started to record their demographic data, and 26 (81.3%) also moved on to select the variables for the score. Twenty-five (64.1%) of the 39 gastroenterologists completed the whole survey and were included in the study. The excluded gastroenterologist provided answers for one single indication and then aborted the questionnaire. The reason for the dropouts cannot be determined due to the study format.

The mean age of the included gastroenterologists was 48.5 years (range 35–69). Mean experience as a licensed gastroenterologist was 9.6 years (range 1–33). Sixteen gastroenterologists (64%) repeated the questionnaire 6 weeks later to test for reliability. The characteristics of these gastroenterologists were not significantly different from the nine who did not repeat the survey. The ICC for the reliability of the sum of the scores for the variables was excellent (0.88 to 0.93) for all the indications. The κ values for the reliability of final scores showed a good to very good agreement in all of the indications. The resulting TPS for all nine indications is presented in online supplementary appendix 1. It consists of 15 items for each indication. Depending on the value of the item for the quality of the referral, it is awarded with 1, 2 or 3 points if described adequately in the referral letter. The maximum score for a referral is 30 points, indicating a high-quality referral letter.

The referring physicians were on average 47.1 years old (range 26–72), 37.3% were female and 95% were GPs. The referred patients were 62.4% female, and the average age was 57.2 years (range 6–94).

Adherence to the Norwegian referral guidelines varied substantially, as shown in table 1.

In particular, information regarding allergies/critical information, family history of disease and alcohol/tobacco consumption was sparse.

The average quality of the referral letters assessed by VAS was 4.7 (95% CI 4.5 to 5.0, range 0.2–9.5). The mean TPS for all referral letters was 13.2 (95% CI 12.8 to 13.8, range 1–25). In total, 54.1% of the referral letters had a VAS below 5 (out of 10) and 59.6% had a TPS below 15 (out of 30).

Intra-rater and inter-rater reliability of the TPS for scoring referral letters was excellent (ICC=0.87 (95% CI 0.81 to 0.92) and 0.91 (95% CI 0.86 to 0.94), respectively).

The average VAS and TPS for the nine indications are shown in table 2 together with the correlation between the two scores. The VAS and the TPS showed a moderate overall correlation (r=0.42; figure 1), ranging from fair to substantial (r=0.24 to 0.63) for the different indications.

Table 2

Referral information quality assessed by VAS and TPS, and correlation between them

Indication	N referral letters (%)	Mean TPS (95% CI)	Mean VAS (95% CI)	Correlation coefficient* (95% CI)	ICC† average measures (95% CI)
Abdominal pain	50 (15.3)	12.5 (11.0 to 14.1)	4.5 (3.9 to 5.2)	0.46	0.49 (0.09 to 0.71)
Dyspepsia	47 (14.4)	11.9 (10.6 to 13.1)	4.3 (3.7 to 4.9)	0.25	0.33 (−0.02 to 0.63)
Gastrointestinal bleeding	34 (10.4)	15.7 (14.1 to 17.3)	5.1 (4.4 to 5.9)	0.46	0.51 (0.03 to 076)
Change of bowel habit	48 (14.7)	14.9 (13.5 to 16.3)	5.1 (4.3 to 5.8)	0.60	0.66 (0.40 to 0.81)
Diarrhoea	38 (11.6)	11.3 (9.9 to 12.7)	4.6 (4.0 to 5.2)	0.37	0.42 (−0.11 to 0.70)
Dysphagia	36 (11.0)	11.9 (10.3 to 13.5)	5.0 (4.3 to 5.8)	0.29	0.37 (−0.24 to 0.68)
Constipation	27 (8.3)	13.5 (12.1 to 14.9)	4.5 (3.8 to 5.1)	0.24	0.30 (−0,55 to 0.68)
Weight loss	21 (6.4)	14.9 (12.5 to 17.2)	4.8 (3.9 to 5.8)	0.63	0.62 (0.06 to 0.85)
Jaundice	26 (8.0)	14.3 (12.4 to 16.2)	4.9 (3.9 to 5.8)	0.24	0.32 (−0.52 to 0.70)
Total	327	13.2 (12.8 to 13.8)	4.7 (4.5 to 5.0)	0.42	0.47 (0.34 to 0.57)

*Pearson correlation coefficient interpretation: 0–0.2=slight, 0.2–0.4=fair, 0.4–0.6=moderate, 0.6–0.80=substantial, 0.8–1.0=almost perfect.

†ICC interpretation: >0.75=excellent, 0.40–0.75=fair to good and <0.40=poor.

ICC, intraclass correlation coefficient; TPS, Thirty Point Score; VAS, visual analogue scale.

Figure 1

Correlation between TPS and VAS

A multiple linear regression analysis with a manual backward elimination procedure showed that age and gender were the only patient-related or doctor-related variables associated with TPS (age: βadj=−1.156, 95% CI −2.24 to (−0.076)), p=0.036; gender: βadj=−0.090 (95% CI −0.136 to (−0.043), p<0.001), explaining 7% of the variance of TPS (r2=0.07). Further, gender was identified as the only variable associated with VAS (βj=−0.513, 95% CI −0.993 to (−0.033), p=0.036), explaining 1% of the variance of VAS (r2=0.01).

When the gastroenterologist had recorded that the referral letter contained too little information (n=167 (51.1%)), the VAS and the TPS were also significantly lower (mean difference (Δ)=1.7, p<0.001 and Δ=3.4, p<0.001, respectively). When the gastroenterologists had recorded that the referral letter was unstructured (n=60 (18.3%)), the VAS was significantly lower (Δ=1.6, p<0.001), but the TPS was unaffected (Δ=−0.4, p=0.51).

There were significant differences in the TPS and the VAS between the centres, and this difference was confirmed by the ANOVA analysis for the TPS (p<0.001) and the VAS (p=0.004). For the TPS, this significant difference disappeared by removing centre III from the calculations. The difference in the VAS disappeared by removing centre II (table 3).

Table 3

Relationship between TPS and VAS by centre

Centre	N (%)	Mean TPS (95% CI)	p Value*	Mean VAS (95% CI)	p Value*
Centre I	45	14.2 (12.7 to 15.7)		4.5 (4.0 to 5.1)
Centre II	42	10.9 (9.5 to 12.4)		4.7 (4.0 to 5.4)
Centre III	24	12.0 (10.1 to 14.2)		6.3 (5.4 to 7.2)
Centre IV	46	13.0 (11.7 to 14.2)		5.5 (5.2 to 5.9)
Centre V	67	13.9 (12.9 to 15.0)		4.2 (3.8 to 4.6)
Centre VI	50	14.7 (13.2 to 16.3)		4.4 (3.6 to 5.2)
Centre VII	53	13.1 (11.8 to 14.3)		4.6 (4.0 to 5.3)
Total	327	13.3 (12.8 to 13.8)	0.004	4.8 (4.5 to 5.0)	<0.001

*ANOVA.

ANOVA, analysis of variance; TPS, Thirty Point Score; VAS, visual analogue scale.

Some analyses were made to identify factors that could increase the correlation between the VAS and the TPS. Eliminating the one-point items did not improve the correlation. Neither did adjusting for the number of three-point items in the referral.

Discussion

Overview and principal findings

This is, to the best of our knowledge, the first study to develop an objective, reliable and validated score (TPS) to assess the quality of gastroenterology referral letters, and it may work as a model for other medical specialties. The score is useful for the majority of referrals in gastroenterology, regardless of the seriousness of the condition or the location of the disease.

The TPS has demonstrated an excellent intra-rater and inter-rater reliability as well as a moderate correlation between the TPS and a subjective VAS score assigned by gastroenterologists. The quality of the referral letters was variable, both assessed by the TPS and the VAS.

The correlation and agreement between the TPS and the VAS was somewhat lower than expected (r=0.42, ICC=0.47). Factors not captured by the TPS may also influence the subjective assessment of the quality of referral letters, such as lack of structure or appropriateness. Such factors may have negatively influenced the correlation between the two measurements. Unstructured letters had a lower VAS despite adequate content according to the TPS.

There were significant differences between the centres both for the mean TPS and the mean VAS. Differences in quality of referral letters in different geographical regions have in other studies been explained by GP workload, referral culture or capacity in local nursing and care institutions, and some of these factors may also be present in this study.23

The only patient-related or physician-related factor associated with a change in quality of the referral letters in this study was increasing age and male sex of the referring physician, both leading to small decreases in the TPS, but the changes are minor and most likely not clinically relevant.

The TPS consists of 15 items for each indication. This number may be too high, as many referral letters may contain sufficient information with fewer items.31 However, eliminating the one-point items did not improve the correlation between the VAS and the TPS.

Also, scoring instruments with many interpretive questions could have a lower reliability.32 Since the TPS demonstrated excellent intra-rater and inter-rater reliability, this does not seem to be an issue with the TPS.

Some score items that may seem of limited relevance for a given indication (eg, information about Faecal Occult Blood Test (FOBT) in the dyspepsia indication) may have been selected by the gastroenterologists due to the variable's ability to discriminate between serious and less serious diseases.

In conclusion, the TPS measures the quality of the information in the referral letter objectively but is not a perfect tool for assessment of overall quality of the referral letter, as this also involves consideration of indication and structure of the referral.

Comparison with the existing literature

A score to assess referral letters in colorectal surgery has previously been suggested by Jiwa et al.33 This score was developed by colorectal surgeons and GPs, and was validated against the likelihood of detecting colorectal cancer, not against any ‘gold standard’ for referral letter quality or on a wider range of gastrointestinal conditions. Consequently, it is not possible to determine whether high scores actually reflect a high quality of referral letters. Thus, for assessing referral letters in general, the TPS is a more useful tool.

VAS is a recognised tool for assessing pain,34 and has been shown to be well suited for assessing soft data in clinical trials.35 It has also been used previously in quality score validation studies.28 We therefore chose VAS as a method to assess the overall quality of the referral letters. Others have also used VAS as a tool to assess quality of referral letters,16 and showed scores similar to the findings in our study, with values between 1.1 and 6.9 for the various information items. However, VAS is a subjective measurement, and cannot replace objective scoring methods for evaluating quality.

Other more general scores for content of referral letters have also been created,7 20 but have given little insight into the specific symptom-related items mentioned in the referral letters, and cannot be used to assess the information quality of the referral letter. Also, referral letters in Norway are mainly generated electronically within the general national referral template,36 and these general scores could consequently indicate a good referral letter, regardless of the description of the patient's symptoms and signs.

Strengths and limitations of the study

An important strength of the present study is the multicentre design, giving the results a higher external validity. This aspect is also maintained by the wide variety of indications included, covering most of the reasons for referrals to Norwegian gastroenterology units.

The inclusion of a large number and wide variety of clinical gastroenterologists in the development of the TPS also ensures that the score reflects what the specialists actually need to effectively assess the referral letters. Further, the comparison of the TPS with the gastroenterologists' subjective assessment of the quality of the referral letter (VAS) enhances the emphasis on what the actual assessors value in the referral letters.

Our study has some potential limitations. First, we have not assessed referral appropriateness and cannot distinguish appropriateness as a deciding factor for lack of correlation between the VAS and the TPS. It is possible that appropriateness of the referral has influenced the VAS, particularly when referrals with poor indication have been well written in terms of clinical information and structure.

We also observed a difference in the VAS between the centres, and it is possible that systematic differences in the use of the VAS may have influenced the results of the study.

In this study, gastroenterologists determined the optimal content of referral letters for easy assessment and prioritisation. A score based on risk factors for gastrointestinal cancer could have been an alternative approach. However, the aim was to develop a score that reflects what makes a referral letter easier to assess and prioritise, not to identify high-risk patients.

Another weakness of the study may be a selection bias for the gastroenterologists who participated in the score development. Thirty-nine gastroenterologists were invited to participate, and the 25 who completed the study may differ from the 14 who did not. However, willingness to participate probably does not influence the validity of their opinions regarding the content of referral letters, and could also be seen as a strength, as an interest in the topic may indicate a better understanding of what should be considered important clinical information in referral letters.

Some of the items selected for the score may be somewhat too unspecific, or may be considered inappropriate for the indication by other gastroenterologists. We have, for example, chosen the unspecific term ‘previous radiology’ in the jaundice/elevated liver enzymes indication, while ‘previous ultrasound of the liver’ may be a more appropriate and specific item. We have also chosen the term ‘current medical treatment’ as we believe this term accounts for any relevant information regarding the patient's medication, including ingestion of anticoagulants or antiplatelet agents.

Implications of the study

We have developed and validated an objective and reliable score for assessing the quality of referral letters in gastroenterology.

The moderate quality of referral letters observed in this study suggests that a tool to facilitate creation of high-quality referral letters would be beneficial. Information technology, like checklists or clinical decision-making systems, may be a part of the solution.

Unanswered questions/future research

This study is a presentation of the TPS that resulted from the survey among the 25 consultant gastroenterologists. However, some refinement of the TPS may be warranted and may increase the ability of the TPS to discriminate between high-quality and low-quality referral letters. Validation of the TPS in other healthcare systems is also necessary to reach a TPS of a high general validity.

In this study, we found a high prevalence of referral letters with scores below the middle value of the scales for both TPS and VAS. Future research should aim to implement and evaluate interventions to improve quality of referral letters in a way acceptable to referring GPs and specialists in the hospitals. A Cochrane review of interventions to improve referrals identified active involvement of secondary care specialists, and implementation of structured referral sheets as the only interventions with effect on referral quality.37 Electronic referrals are the norm in Norway, and the implementation of structured referral sheets/checklists in electronic referrals may be an interesting intervention to explore.

Conclusion

The TPS is reliable, objective and shows good agreement with the subjective VAS. The score may be a useful tool for assessing referral quality in gastroenterology, particularly important when evaluating the effect of interventions to improve referral quality. The method used in the development of the score can serve as a model for other medical specialties.