Characteristics of Exceptionally Good Doctors: A Protocol for a Cross-Sectional Survey of Adults.

Background: Doctors constitute a significant proportion of a very large number of medical interactions. They are known to vary in the quality of their work, with some having an exceptionally beneficial effect on patients' physical health. In a qualitative study, we interviewed medical doctors on their opinions and experiences of exceptionally good doctors. Their responses and the results from previous research are used as a basis for this proposed cross-sectional survey directed to members of the public on their encounters with exceptionally good doctors. The primary aim of this cross-sectional study is to describe the characteristics of exceptional doctors as reported by a large representative sample of adult patients. Methods and Analysis: A mixed qualitative and quantitative anonymous cross-sectional survey of 500 Amazon Mechanical Turk (MTurk) respondents, who have met one or more exceptionally good doctors in their life, will be conducted. Information requested will include reasons for nominating a particular doctor; experience of how that doctor differs from other and average doctors; and 34 5-point Likert scale questions on the characteristics of that doctor and the same Likert questions for the average doctor. An opportunity to report their experience in free-text form will be provided. Sample size will be sufficient to obtain a margin of error of 4%. The authors will provide descriptive statistics, including graphs of the Likert scale question responses; conduct factor analysis for internal validity; investigate satisficing and logical inconsistencies; and explore whether there are multiple types of exceptionally good doctors. Discussion: Previous surveys of patients' perceptions of doctors exist though none have focused on exceptionally good doctors. The expected results will include a list of characteristics that are important to patients in determining exceptionally good doctors.

Plain Language Summary

Previous research has shown that some doctors are exceptionally good. In a qualitative study, we have interviewed doctors about what characteristics make up an exceptionally good doctor. Using the interview results, we have designed a survey to investigate the opinions of the general adult public on exceptionally good doctors. The survey will be conducted via Amazon’s Mechanical Turk (MTurk) platform, where members are offering to provide online services like answering surveys or identifying objects in images. MTurk has been used successfully in previous research surveys. The responses to this survey will provide a list of the important characteristics of an exceptionally good doctor from a patient perspective.

Background

“[T]he most frequently used drug [intervention] in general practice is the doctor himself”.1,2 In Australia, general practitioners receive more than 150 million visits per year.3 That is an average of six visits for each Australian. Adding visits to specialists and extending this to the entire world, we can expect billions of times this intervention, “the doctor himself” is used each year.

It is possible to identify doctors who have an exceptionally beneficial impact on patients’ physical health, even after accounting for all known confounding factors such as patient risk, doctor demographics, and hospital factors.4–16 Evidence from systematic reviews has shown that surgical experience, measured in case volume or years of practice, is a key factor in improved surgical performance and outcomes, supporting the need for a continuing learning process.17,18 Provider expectations can substantially affect patient outcomes.19 For non-surgical specialties, data have suggested that doctor-related factors have a considerable effect on patients’ health outcomes in primary,20 acute care,21 and obstetrics.10 Abu-Hilal et al22 studied the characteristics of good doctors, concluding that clinical ability, knowledge, approachability, and appreciation of limitations are the most important factors. Qualitative study research has identified eight vital skills of exceptionally good doctors in their relationships with patients: “do the little things; take time; be open and listen; find something to like; remove barriers; let the patient explain; share authority; and be committed”.23 Steiner-Hofbauer et al's24 systematic review of “good doctors” included 6 studies and 2 questionnaires,25–29 and concluded that there is no clear definition on what makes a good doctor.

Although there are published opinions on what makes a good doctor,30–40 the character of a good doctor,41 divergent patient and doctor viewpoints,29 and a thesis on the good doctor in education,42 exceptionally good doctors have been investigated infrequently.23,43,44 It is not known what makes an exceptionally good doctor.

In a qualitative study, we have interviewed a sample of medical doctors on their opinions of what makes an exceptionally good doctor.45,46 These interviews have led to a set of characteristics that describe an exceptional doctor from the practitioner point of view.

A systematic review on the links between patient experience and clinical safety and effectiveness reported “consistent positive associations between patient experience, patient safety and clinical effectiveness for a wide range of disease areas, settings, outcome measures and study designs”47 and therefore “supports the case for the inclusion of patient experience as one of the central pillars of quality in healthcare.” Patient tutors now work in collaboration with clinical staff to develop the curriculum and assessment of medical students in the UK.48

Given the importance of patient input into medical education, the positive associations between patient experience and clinical effectiveness, and the knowledge gap on what makes an exceptional doctor, we plan to conduct a survey of adult patients on their experiences with exceptionally good doctors. The primary aim of the survey is to describe the characteristics of exceptional doctors from a patient perspective.

Methods and Analysis

The study design reporting follows the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) guidelines on observational studies.49

Study Design

This is an observational cross-sectional study, using a mixed qualitative and quantitative anonymous online survey design to collect data on patients’ perceptions of exceptionally good doctors. In particular, this is both an exploratory sequential design mixed methods study50 as the authors previously interviewed 13 medical doctors about exceptionally good doctors45,46 and used the insights gained to produce the quantitative part of this survey. It is also a convergent design mixed-method study50 as the three qualitative questions and six questions with a free-text option “other” provide a more in-depth and personal perspective of the respondents.

Participants will be sampled through Amazon Mechanical Turk.51

Data will be collected for the respondent demographic information on gender; age by decade; highest education level achieved; count of doctor visits in the previous year; and country will be deduced from the respondents’ IP address. The respondents will be asked how many exceptionally good doctors they have met in their life (1, 2, 3, 4, 5+) and how many overall doctors they have met (1–5, 6–10, 11–20, 21–50, 51–100, 101+).

Questions asked in the survey about the nominated exceptionally good doctor are the doctor’s age by decade, gender, type of doctor (GP, specialist, other), the respondent’s relationship to that doctor (patient, employer, etc), and the reason for nominating this doctor as exceptionally good, with the reasons offered for the respondents to choose from falling into the broad categories of firstly being an exceptional person or behaving exceptionally by being an exceptional communicator or empowering patients, secondly being an exceptional diagnostician or, thirdly, being an exceptional wielder of interventions. A further question asks how many characteristics are needed for a doctor to be exceptional.

In addition there are 34 5-point Likert scale questions. The Likert scale questions were chosen based on responses from a sample of doctors interviewed as part of a qualitative study on the characteristics of exceptional doctors.45 The full list of questions is shown in the .

With the Likert questions, the respondents are asked how much they agree with the nominated doctor having specific traits compared to an average doctor.

A further qualitative question offers the respondents the opportunity to relate their experience with the exceptionally good doctor into their own words.

One question displays all the Likert question items where the respondent gave 4.5 or more out of 5 for the exceptionally good doctor, and asks to pick the top three such questions and rank them.

Participants

The participants will be recruited from Amazon’s Mechanical Turk (MTurk) population. As per Amazon’s website,51 “Amazon Mechanical Turk (MTurk) is a crowdsourcing marketplace that makes it easier for individuals and businesses to outsource their processes and jobs to a distributed workforce who can perform these tasks virtually. This could include anything from conducting simple data validation and research to more subjective tasks like survey participation, content moderation, and more.

MTurk has been used by thousands of social scientists in research52 and is very suitable for surveying the general population.53–55 In addition, the large number of MTurk workers alleviates concerns of non-naivety of the workers.56 MTurk works by logging in if you have an Amazon account, or by registering as a requester. The requester creates a new project by providing instructions and a link to the survey, to which the workers are requested to respond, in this case a Qualtrics57 survey. The Qualtrics survey creates a code that is displayed to the worker if they finish the survey, which should be entered by the respondent into the Amazon MTurk instruction page, as proof of them having done the survey. Thereafter, the requester is enabled to download a list of the workers’ anonymous IDs and the codes the workers have submitted, compare this list to the codes that were generated, include those who submitted a correct code and exclude those who did not.

The demographics of MTurk workers are generally representative of the population, except that they are skewed younger,58 therefore, to assure a representative sample, we conducted a pilot study of 210 respondents, of which 6% were found to be over 55, and 67% were males. To obtain a gender split that is closer to 50/50 and to have a representative age-based distribution, the authors will submit four requests, also known as batches, as per Table 1. In this manner, we expect to have around one third of participants over 55 and approximate gender parity. MTurk makes it possible to ensure that no worker participates in more than one project or batch. Survey weighting59,60 will be used for age and gender if the respondents’ age and gender are more skewed than expected.

Table 1

MTurk Batches

Participants	Batch	Size of Batch	Expected % Female	Expected Count of Females	Expected % >55	Expected Count of Aged >55
Any demographic	1	250	33	83	6	15
Female, any age	2	101	100	101	6	6
Any gender, aged >55	3	125	33	42	100	123
Female, aged >55	4	24	100	24	100	24
		500	50%	250	34%	168

Sample Size

We have based our sample size of 500 participants on a desire to obtain a margin of error of 4% around estimates of proportions of participants reporting particular characteristics of exceptional doctors (eg, communication skills) and reasons why they consider the doctor in question to be exceptional (eg, they saved my life).

Bias

Selection Bias

Since MTurk population tends to be skewed towards younger-aged workers and males, a random sample could lead to sampling bias. We will avoid this possibility by selecting a stratified sample to ensure 34% of the population being aged 55 years or older and gender equality.

Nonresponse Bias

In the MTurk marketplace, where the survey is accessed by Amazon’s workers, it is not possible to determine the number of potential respondents who have seen the survey but chose not to participate. Consequently, it is not possible to accurately calculate the response rates. Nevertheless, it is argued that response representativeness is more important than response rate in survey research.61 Therefore, since we will attempt to obtain a representative sample, this bias will be less of a concern.62

Response Bias

Question order bias will be addressed by randomizing the order in which the response categories within the Likert question sets are presented.

In addition, there is a potential for bias as the questions have been derived from interviews with medical doctors on what they consider to be the characteristics of exceptionally good doctors. Any putative exogenous characteristics that pertained to exceptionally good doctors were addressed in the pilot study by offering the respondents a free-text opportunity to add further characteristics; likewise, several free-text spaces have been added to the final version of the survey.

Missing Data

All questions, except for the free-text questions, will be mandatory; therefore, no missing data is expected. However, since this requirement may lead to satisficing, ie respondents not replying truthfully in order to get to the end of the survey more quickly,63 the responses will be checked for unusual patterns such as many identical responses to the Likert questions.

Confounding variables are not an issue as no cause-and-effect relationship is investigated, only opinions and experiences.

Planned Analysis

The authors will provide descriptive statistics of the demographic variables supplied, in addition to the combined graphs of the Likert questions. The 10 most commonly reported characteristics of exceptional doctors will be reported as proportions with 95% confidence intervals. The frequency of reasons why a doctor was nominated as exceptional will also be reported as proportions with 95% confidence intervals.

Factor analysis64 of the Likert questions in the pilot study showed that there were 37 factors, in which component one had an eigenvalue of 18, component two had an eigenvalue of 1.1, while all other components had smaller eigenvalues. Thereafter, as factors 22–37 were very slightly negative in their contributions, the first factor accounted for 82% of the total proportion. Therefore, the pilot survey overwhelmingly measured a single factor: the characteristics of exceptionally good doctors.

The validity of the Likert questions for identifying exceptionally good doctors will be further explored by asking the same Likert questions in relation to average doctors and then assessing whether factor analysis identifies separate factors for exceptionally good doctors and average doctors. Further, the Likert questions will be specified as dependent variables in linear regression models to explore whether the demographic and doctor characteristics show an association with the Likert question responses. As the Likert question responses may not follow a normal distribution, nonparametric regression will also be conducted.65

To facilitate visual comparisons between the responses to the Likert questions, “[k]ernel density estimates [plots], which can be considered as a smoothed form of histogram”66 will be constructed. Kernel density plots allow multiple histogram outlines to be shown in a single graph.

Subgroup comparisons will be performed for all relevant variables to explore whether responses differ by age group, gender, education levels, or count of doctor visits.

The free-text question 5 responses will be analyzed to investigate whether the traits, aspects or qualities that make a doctor exceptional listed by the respondents at the very beginning of the survey are consistent with the responses provided by the 13 medical doctors interviewed in the qualitative study. The free-text question asking respondents for their personal experience will be analyzed by the usage of an NVivo67 word cloud to detect patterns.

Ethical Considerations

Ethical approval has been received from Bond University. A preprint shows that it is ethical to use MTurk as a survey tool.68 An informed consent form will be provided at the beginning of the survey. The participants will be informed that they can stop the survey at any time though they will forego payment if they do. The data will be stored at Bond University in a secured facility by the lead researcher on a password-protected computer.

Discussion

There is existing research on patients’ opinions on what constitutes a good doctor,69–71 but not on what constitutes an exceptionally good doctor. Currently, there is no clear definition on what is an exceptionally good doctor. A qualitative study aims to provide a set of characteristics that describe exceptional doctors from the doctor perspective.45 This proposed study will add to this knowledge base by providing a set of characteristics that describe exceptional doctors from the patient perspective.

Strengths

A strength of this proposed survey is that the questions are based on in-depth interviews with medical doctors on what they consider to be the characteristics of exceptionally good doctors. The survey also has a qualitative element to allow the respondents to report their experience in a free-text format. The participant responses to the survey questions will be important to anyone in healthcare practice and medical education but also of interest to the general public.

Limitations

A limitation is that this is a survey of respondents who speak English and are mostly from developed nations with a preponderance of respondents being from the US. Therefore, it is uncertain whether the findings could be generalized to other regions, particularly developing nations. In addition, due to the heterogeneity regarding medical/surgical specialties, types of interventions, and types of outcomes relevant to different interventions, there may be differing criteria on what makes an exceptionally good doctor for different medical/surgical specialties.