The SIMARD-MD is not an Effective Driver Screening Tool for Determining Fitness-To-Drive.

BACKGROUND: Studies have reported poor sensitivity and specificity of the Screen for the Identification of Cognitively Impaired Medically At-Risk Drivers, a modification of the DemTech (SIMARD-MD) to screen for drivers with cognitive impairment. The purpose of this study was to determine whether the SIMARD-MD can accurately predict pass/fail on a road test in drivers with cognitive impairment (CI) and healthy drivers.
METHODS: Data from drivers with CI were collected from two comprehensive driving assessment centres (n=86) and compared with healthy drivers (n=30). All participants completed demographic measures, clinical measures, and a road rest (pass/fail). Analyses consisted of correlations between the SIMARD-MD and the other clinical measures, and a receiver-operating-characteristic (ROC) curve to determine the predictive ability of the SIMARD-MD.
RESULTS: All healthy drivers passed the road test compared with 44.2% of the CI sample. On the SIMARD-MD, the CI sample scored significantly worse than healthy drivers (p < .001). The ROC curve showed the SIMARD-MD, regardless of any cut-point, misclassified a large number of CI individuals (AUC=.692; 95% CI = 0.578, 0.806).
CONCLUSIONS: Given the high level of misclassification, the SIMARD-MD should not be used with either healthy drivers or those with cognitive impairment for making decisions about driving.

INTRODUCTION

Approximately one quarter of the Canadian population will be aged 65 years and older in the next 10–20 years.( The importance of driving a personal vehicle for maintaining independence and autonomy, along with increasing life expectancy, has resulted in seniors keeping their driving licences well into their 80s.( However, with increasing age also comes an increase in age-related medical conditions, such as cognitive impairment/dementia, that can impact the ability to drive safely.( Given that projections in Ontario alone show there will be close to 100,000 drivers with dementia by 2028,( there is increasing pressure for health-care professionals and licensing authorities to screen for drivers with cognitive impairment/dementia who may be unfit to drive.

Cognitive impairment (CI) is present when a person has difficulties with memory, learning new skills or things, and concentration or decision-making that can impact instrumental activities of daily living such as driving. Compared to healthy older drivers, research shows that drivers with dementia are more likely to fail a road rest( and have a 2–3 times higher crash risk.( However, some studies show that drivers with early stage dementia can continue to drive safely,( although the progression of cognitive impairment/dementia will likely require eventual licence revocation.

Identifying when drivers with cognitive impairment/dementia have become unsafe remains a challenge for health-care professionals. One tool in particular, the Screen for the Identification of Cognitively Impaired Medically At-Risk Drivers, a modification of the DemTech (SIMARD-MD), has been controversial in its recommendation for use by frontline clinicians. The SIMARD-MD was designed to identify cognitively-impaired drivers whose driving skills may have declined to an unsafe level.( As indicated in the title of the instrument, the SIMARD-MD is derived from the DemTect.( The DemTect was developed to measure cognitive impairment and includes five sections: word list recall, a number transcoding task, a semantic verbal fluency task, reverse digit span, and delayed recall of the word list. However, the DemTect was not developed to identify unsafe drivers.

While the authors of the SIMARD-MD asserted it has a “high degree of accuracy” and “can be used for immediate decisions in the clinical setting”( to identify unsafe drivers, the presumed value of the tool has been questioned since the manuscript was published.( In the initial study with a sample of 146 cognitively impaired seniors, the authors found that the SIMARD-MD predicted 86% and 84% to fail and pass a road test, respectively.( A validation study with 192 cognitively impaired seniors similarly found that the SIMARD-MD predicted 80% and 87% of those predicted to fail and pass a road test, respectively.(

However, these numbers indicate a substantial number of false-positive and false-negative test results. Furthermore, the sample included a number of cognitively healthy drivers—introducing spectrum bias—and, in reporting the results above, the authors did not include drivers identified as “indeterminate” and requiring further testing (about 50% of the sample).( Ignoring half of one’s sample in determining the predictive value of a test renders the results meaningless within a screening context. A stronger presentation of their data would have included a receiver-operating characteristic (ROC) curve to allow the determination of optimal cut-points and the associated sensitivity and specificity. In a different study, Bédard and colleagues( concluded that the SIMARD-MD was vulnerable to an education bias, and may create a structural inequity by failing to account for education in the interpretation of the test results. Moreover, Wernham and colleagues( found no association between SIMARD-MD scores of clients with cognitive impairment and the clinical impressions of their geriatricians regarding fitness-to-drive, further questioning the validity of the test results.

These studies are particularly relevant given that the SIMARD-MD has been suggested to health-care professionals as a tool to determine fitness-to-drive in Canada and appropriate for routine administration in the family physician office setting once patients reach age 70.( Yet, despite the availability of the SIMARD-MD and strong claims about its value, there is a paucity of research on its clinical utility. There are legitimate concerns regarding the predictive ability of the SIMARD-MD towards pass/fail outcomes on road tests and its value within a clinical setting.

Hence, the objective of the present study was to evaluate the utility of the SIMARD-MD as a screening tool in a sample of CI older adults and in a sample of cognitively healthy older adults. Specifically, we compared SIMARD-MD scores between CI drivers and healthy controls, examined the extent to which the SIMARD-MD correlated with other common clinical measures, and determined whether SIMARD-MD scores differentiated between pass/fail outcomes on the road test in CI drivers. Additionally, we determined the ability of the SIMARD-MD to accurately predict pass/fail outcomes in CI drivers.

METHODS

Setting

This study received ethics approval from the University of Waterloo, University of Saskatchewan, and Lakehead University. Data from comprehensive driving evaluations, performed by an occupational therapist, were collected retrospectively from one health centre in southwestern Ontario and from one health centre in Saskatchewan. Comprehensive driving evaluations consist of a clinical battery of tests assessing vision, motor control, and cognition, followed by a road test.

Given that comprehensive driving assessment centres typically test drivers referred for medical reasons, cognitively healthy drivers were recruited to provide a comparison group. Cognitively healthy participants were recruited from a city in northwestern Ontario through posters or direct telephone/e-mail solicitation of individuals who had previously consented to be contacted for driving research. These participants provided informed consent and completed a clinical assessment and a road test.

Participants

During the study time period there were 201 drivers from southwestern Ontario and 183 drivers from Saskatchewan who were referred for a comprehensive driving evaluation (CDE) either by the licensing authority or a physician. Of these drivers, 88 had cognitive impairment and/or dementia and were referred by a physician.

Thirty cognitively healthy participants were recruited and met the following conditions: 1) were community dwelling; 2) aged 70 years or older; 3) fluent in English, 4) possessed a valid driver’s licence, and 5) drove at least once per week. The exclusion criterion was a Standardized Mini-Mental State Examination (SMMSE) score of less than 24 (no participants scored below this threshold). All cognitively healthy participants received a $10 gift card for their participation.

Clinical Measures

Drivers provided basic information about their driving history. Participants also completed the SIMARD-MD, the Trail A and B tests, and the Useful Field of View (UFOV) test.

The SIMARD-MD test includes two subtests assessing immediate recall and one subtest measuring delayed recall, two tasks that measure the ability to remember words, one task involving converting numbers to words, and one task where the participant names objects that can be purchased at a supermarket within a one-minute time frame. Scores can range from 0 to 130; higher scores indicate better cognitive abilities. The SIMARD-MD score is classified into one of three categories: ≤30, 31–70, and >70. According to the test developers, a score of ≤30 predicts that the person will fail a road test, from 31 to 70 it is uncertain whether they will pass or not, and those who score higher than 70 are predicted to pass a road test.( The Trail Test A and B tests are used to assess psychomotor speed and divided attention, respectively.( Individuals with longer completion times on the Trails B, in particular, have greater odds of failing a driving test (OR = 2.5, 95% CI: 1.0–5.9).( The UFOV( includes three distinct parts measuring (in milliseconds) processing speed, divided attention, and selective attention, and is associated with on-road performance; higher scores indicate poorer performance.( Based on the completion times of the three UFOV sub-tests, a composite score from 1 to 5 is produced; higher scores indicating greater risk for poorer driving performance (1 = very low risk, 2 = low risk, 3 = low–moderate risk, 4 = moderate–high risk, and 5 = high risk).

The Road Test

The road test for the CI sample required participants to drive for 45–60 min, depending on traffic patterns. The test was conducted during the day in good weather conditions and included driving on residential, suburban, urban and expressway sections. The occupational therapist determined participants’ driving performance and provided ratings for pass, fail, or fail with lessons and retest. The two fail categories were collapsed and a binary variable was produced for statistical analyses (pass vs. fail). Pass/fail determinations were made using a weighted scoring system based on a total of 95 manoeuvres. A percentage score, based on the correct number of manoeuvres performed on different driving tasks (yielding, signalling, vehicle position, lane maintenance, scanning, gap acceptance speeding), was calculated. Any driver scoring less than 47.5 (or correctly performing less than 50% of all manoeuvres) failed the road test. Additionally, any driver who made hazardous driving errors (e.g., running a stop sign, crashing) received a fail outcome.

A driving instructor with 20 years of experience, from a driving school approved by the Ministry of Transportation of Ontario (MTO), conducted the road tests in cognitively healthy participants. The instructor was blind to the results of the clinical tests. The road test and scoring approach were designed to meet the requirements of the MTO for driver licensing examinations. Each road test was conducted using the same vehicle with dual pedals, using a standardized circuit taking approximately 45 min. Participants received a score out of 100; higher scores indicated better driving performance. A score of 70 or greater was considered a “pass”; scores below were deemed a “fail”. However, serious errors (e.g., which may have resulted in a crash) would have resulted in an automatic “fail” regardless of the final score. All drivers were tested during the day in good weather conditions. For all drivers, the driving instructor also provided an assessment on whether they would benefit from additional training.

Data Analysis

Data (demographic information, clinical scores, and road test results) were entered into a SPSS database (version 26.0) by graduate research assistants. Data entry was monitored by the research team to ensure data completion and accuracy. Descriptive statistics were calculated for demographic information (e.g., age and gender), clinical measures, and the road test. Continuous variables are shown using the mean and standard deviation, while categorical variables are presented using frequencies and percentages. Pearson correlations were conducted to determine associations between SIMARD-MD scores and the other clinical measures (Trails A and B, UFOV). Independent t-tests or Chi-squares tests were used to examine the association between clinical scores and pass/fail outcomes on the road test. Two-tailed significance tests with alpha ≤.05 were used for all analyses.

A ROC curve was used to determine the validity of the SIMARD-MD against the pass/fail road test results. The ROC curve is a graphical representation of various cut-off points (based on the SIMARD-MD scores), each measuring the rate of true positives (sensitivity) vs. the rate of false positives (1-specificity), at different cut-off points for the SIMARD-MD. The AUC is considered an index of the overall predictive utility of a screening test and ranges from 0 to 1.0 (perfect prediction), where .50 represents chance discrimination, and .70 to .90 is considered an acceptable magnitude.( The ROC curve also captures the predictive value of a positive test (PPV), the predictive value of a negative test (NPV), and error (1-sensitivity + 1-specificity; essentially false negatives + false positives). A ROC curve was only performed on participants with CI due to the genuine uncertainty regarding driving safety for these individuals, and to prevent the introduction of spectrum bias with the inclusion of cognitively healthy drivers.( Using SPSS, ROC curves and AUC estimates, 95% confidence intervals, and p values were generated. We also determined the specificity, sensitivity, PPV, NPV and error rate at selected cut-off points.

RESULTS

Sample Description

Individuals with CI ranged in age from 45 to 94 years (mean 75.2, SD = 10.2); 77% were men. Three quarters of the sample had less than grade 12 education (75%), with the remaining 25% having completed college or university. Comorbid medical conditions most commonly reported were hypertension (23.5%), arthritis (13%), diabetes (10.3%), depression (11.7%), and stroke/TIA (7.4%). The cognitively healthy group ranged in age from 70 to 87 years (mean age of 75.6, SD = 5.1); 56.7% were men.

Prior to the comprehensive driving evaluation (CDE), the CI sample reported a history of driving ranging from 29 to 73 years. About 12% of the sample reported being involved in a crash and 8.4% receiving citations in the past two years. A physician referred all CI individuals for a CDE; five of whom had previously failed a first CDE. Prior to the CDE, the healthy participants reported a history of driving ranging from 39 to 71 years.

Clinical Scores

As shown in Table 1, individuals with CI, compared to cognitively healthy participants, had poorer results on the Trail A and B tests, had poorer processing speed, divided and selective attention on the UFOV, and had worse scores on the SIMARD-MD. When the UFOV and SIMARD-MD were categorized, a significantly greater proportion scored in the impaired ranges.

TABLE 1

Comparison of clinical measures between CI and Control participantsa

Measures	CI Groupn=88	Healthyn=30	Test Statistics
SIMARD-MD Mean Score	35.0 (20.8)	77.2 (20.0)	t = −9.70, p < .001
	2–98	26–114
SIMARD-MD Index
≤30	38 (43.2%)	1 (3.3%)	ç2 = 52.5, p < .001
31–70	44 (50%)	8 (26.7%)
>70	6 (6.8%)	21 (70.0%)
Trails A time (sec)	72 (40)	41 (13)	t = 46.25, p < .001
	23–241	26–77
Trails B time (sec)	288 (159)	102 (40)	t = 9.59, p < .001
	57–786	43–203
UFOV (msec)
Subtest 1	90.5 (145)	32 (18)	t = 3.53, p = .001
	9–500	17–78
Subtest 2	293 (189)	79 (82)	t = 8.13, p < .001
	9–500	17–247
Subtest 3	379 (147)	218 (92)	t = 6.66, p < .001
	16–500	63–417
UFOV Risk Indexb
1 – very low	16 (18.2%)	16 (55.2%)
2 – low	13 (14.8%)	8 (27.6%)	ç2 = 23.6, p < .001
3 – low to moderate	11 (12.5%)	3 (10.3%)
4 – moderate to high	18 (20.5%)	2 (6.9%)
5 – high	30 (34.1%)	0 (0)

For continuous variables, the descriptive statistics shown are the mean, standard deviation (in brackets), and the range. For categorical variables, the descriptive statistics presented are the number of observations and proportions falling in each category. The corresponding statistical tests to compare groups are respectively independent t-tests and Chi-Square tests.

UFOV Risk Index: n=29 for healthy group.

Table 2 shows the correlations between the clinical measures for the CI sample. Almost all clinical measures were significantly correlated with each other. The SIMARD-MD was weakly associated with the Trail Making A test and moderately associated with the Trails Making B test, as well as all three UFOV subtests.

TABLE 2

Correlations between clinical measures in CI drivers (N=86)

	SIMARD-MD	Trails A	Trails B	UFOV 1	UFOV 2	UFOV 3
SIMARD-MD	-	−.27a	−.48b	−.45b	−.49b	−.42b
Trails A	−.27a	-	.28a	.46b	.48b	.47b
Trails B	−.48b	.28a	-	.19	.57b	.60b
UFOV 1c	−.47b	.46b	.19	-	.52b	.41b
UFOV 2d	−.49b	.48b	.57b	.52b	-	.66b
UFOV 3e	−.42b	.47b	.60b	.41b	.66b	-

p<.01.

p<.001.

UFOV Subtest 1: n = 80.

UFOV Subtest 2: n = 75.

UFOV Subtest 3: n = 71.

Road Test Performance

All cognitively healthy participants passed the road test, although 24 (80%) were considered candidates who would benefit from additional training (given the development of bad habits over time, this is possibly true of the general population). Associations between pass/fail outcomes on the road test and SIMARD-MD mean and risk index scores could not be calculated as no participant failed the road test. However, according to the SIMARD-MD scores, 21 of these participants were predicted to pass the road test, eight were in the “indeterminate” category and would require further testing, and one was predicted to fail the road test.

Table 3 compares scores on the clinical measures and pass/fail outcomes on the road test for the CI group. Of the 86 CI drivers, 48 (55.8 %) failed the road test. The SIMARD-MD mean scores were significantly poorer in those who failed the road test (t = 3.31, p = .001); however, there were no significant differences between SIMARD classifications between those who failed and passed the road test (p = .053). Participants with CI who were older and who performed more poorly on the Trail B test and all three UFOV subtests, were significantly more likely to fail the road test.

TABLE 3

Characteristics of CI drivers based on road test results (N=86)a

Measures	Passn=38	Failn=48	Test Statistics
Age	69.1 (11.4)	80.1 (5.8)	t = −5.41, p < .001
	45–88	60–94
Gender	84% male	73% male	ç2 = 1.58, p =.21
Trails A time (sec)	68 (39)	72 (34)	t = −0.42, p =.67
	23–201	30–211
Trails B time (sec)	238(171)	327 (141)	t = −2.51, p =.014
	57–685	97–786
UFOV (time in msec)
Subtest 1	56 (119)	120 (162)	t = −2.01, p =.048
	9–500	9–500
Subtest 2	200 (184)	375 (153)	t = −4.39, p < .001
	9–500	9–500
Subtest 3	310 (146)	444 (117)	t = −4.19, p < .001
	83–500	16–500
UFOV Risk Indexb
1 – very low	12 35.3%)	2 (5.1%)	ç2 = 22.91, p < .001
2 – low	9 (26.5%)	2 (5.1%)
3 – low to moderate	2 (5.9%)	7 (17.9%)
4 – moderate to high	6 (17.6%)	9 (23.1%)
5 – high	5 (14.7%)	19 (48.7%)
SIMARD-MD	43.2±22.11	28.9±17.9	t = 3.31, p = .001
	5–98	2–72
SIMARD-MD Index
≤30	11 (28.9%)	26 (54.2%)	ç2 = 5.84, p = .053
31–70	23 (60.5%)	20 (41.7%)
>70	4 (10.5%)	2 (4.2%)

For continuous variables, the descriptive statistics shown are the mean, standard deviation (in brackets), and the range. For categorical variables, the descriptive statistics presented are the number of observations and proportions falling in each category. The corresponding statistical tests to compare groups are respectively independent t-tests and Chi-Square tests.

UFOV Risk Index: n=73 (n=34 who passed; 39 who failed).

Receiver Operating Characteristic Curves

Figure 1 shows the ROC curve for the CI sample. The AUC is = 0.692, 95% CI = (0.578, 0.806), p = .003. Of the 48 individuals who failed the road test, 26 scored ≤30, 20 scored in the indeterminate range (31–70), and 2 scored > 70. Of the 38 who passed the road test, 11 scored ≤30, 23 scored in the indeterminate range (31–70), and 4 scored > 70. When examining individual cut-points on the SIMARD-MD, none provided both good specificity and sensitivity. For example, using a score of ≤30 provided a specificity of .71, a sensitivity of .54, a PPV of .70, and a NPV of .55. Similarly, using the cut-point of ≤70 provided a specificity of .60, a sensitivity of .67, a PPV of .11, and a NPV of .96. These findings suggest that the SIMARD-MD score of ≤30, which is used to identify unsafe drivers, and scores of >70, which is used to identify safe drivers, misclassify a large number of individuals.

FIGURE 1

ROC curve based on SIMARD-MD scores predicting pass/fail on the road test in drivers with CI. Specificity is defined as the probability that a person who passes the road test (a true negative) has a negative result on the SIMARD-MD (score of ≤ 30, ≤ 40, ≤ 50, etc). Sensitivity is defined as the probability to obtain a positive test on the SIMARD-MD (score of ≤ 30, ≤ 40, ≤ 50, etc) when a person fails the road test (true positive).

DISCUSSION

Although the SIMARD-MD mean scores were statistically different between those who passed and failed the road test, the SIMARD-MD was not predictive, at an individual level, of the pass/fail outcome on the road test. Using the selected cut-points on the SIMARD-MD to predict pass/fail on the road test resulted in a high degree of error, regardless of the cut-point chosen, reflecting the typical trade-off between specificity and sensitivity. This resulted in a large number of misclassifications. For example, 29% of participants who scored 30 or below on the SIMARD-MD passed the road test. Scoring in the indeterminate range was near a 50% chance of passing/failing the road test, similar to the findings of prior studies.( The high misclassification error rate, regardless of the cut-point used, is further evidenced by the limited AUC (.692). The data show that the SIMARD-MD does not have the ability to accurately distinguish between safe and unsafe drivers, and should not be used as a sole screening tool in drivers with cognitive impairment.

The SIMARD-MD was moderately correlated to other measures typically used in comprehensive driving evaluations. Mean scores on the SIMARD-MD were negatively and moderately associated with the Trails B and all three UFOV subtests, as expected. This highlights some degree of overlap between the characteristics measured by the SIMARD-MD and those measured by other tools associated with driving performance. However, the data also show that the SIMARD-MD does not perform any better than other common clinical tests in differentiating between pass/fail outcomes in CI drivers. Specifically, the SIMARD risk indices did not significantly distinguish between CI drivers who passed or failed the road test, whereas the UFOV risk index and Trail Making B test did.

Consistent with other studies, we found that drivers with CI perform significantly worse on measures of visual and cognitive attention compared to healthy participants. Individuals with CI took on average almost 5 min to complete the Trails B and more than two thirds were classified as moderate-to-high risk on the UFOV (Risk Index 3 and higher). Individuals with CI who failed the road test, compared to those who passed, were significantly older and scored more poorly on Trails B, the UFOV, and the SIMARD-MD. However, almost 40% of participants who passed the road test were classified as moderate-to-high risk on the UFOV (i.e., Risk Index of 3 and above) and took 4 min to complete the Trails B, on average. These findings also show that other common measures, such as the Trails B and UFOV, can also misclassify those who should pass or fail a road test.

Similarly, in the cognitively healthy group where all drivers passed the road test, the SIMARD-MD would have incorrectly classified 30% of the participants (eight as indeterminate and one as predicted to fail the road test). This is a good example of problems arising with positive test results in low prevalence situations; when the prevalence of a condition is low, as it was with the absence of unsafe drivers in the cognitively healthy group, the number of false positives (i.e., those scored as indeterminate or fail on the SIMARD-MD when, in fact, they are fit to drive) generally increases. Using a test that has been developed to detect a given condition will not have the same psychometric properties when used with a population that has “different amounts of the trait in question”.( Given that the SIMARD-MD was developed with a sample where half of the participants were deemed unsafe, using it in a clinical setting where there are few unsafe drivers (e.g., a family practice office setting) would result in many false positives. This would lead to an overly large number of drivers requiring further testing, costing millions of dollars in additional road tests that would not be necessary, causing undue stress, and possibly even leading to safe drivers losing their driving privilege.

A limitation of our study was that we did not control for education. While there are no guidelines on the SIMARD-MD for education adjustment,( a prior study found that drivers with post-secondary education scored 8.19 points higher on the SIMARD-MD than those who did not have post-secondary education.( In our study, 75% of CI individuals did not have post-secondary education; these individuals may have been at greater risk of being labelled as indeterminate or likely to fail a road test than participants with more education. Nevertheless, our goal was to test the SIMARD-MD and score it as proposed by its authors. Given our results, it is doubtful that it would perform much better even if there was a small adjustment for education.

We found that using the SIMARD-MD resulted in a large number of false positives and false negatives. Further, given that approximately 50% of the participants fell into the indeterminate range and would require further assessment, it is unclear what value the test really provides. We conclude first that the SIMARD-MD should not be used as a routine test in a family practice office setting. Second, where there are concerns about someone’s ability to drive safely, it must not be used as the sole determinant for driving recommendations, if at all.