Concordance of chart and billing data with direct observation in dental practice.

OBJECTIVES: The commonly used methods of chart review, billing data summaries and practitioner self-reporting have not been examined for their ability to validly and reliably represent time use and service delivery in routine dental practice. A more thorough investigation of these data sources would provide insight into the appropriateness of each approach for measuring various clinical behaviors. The aim of this study was to assess the validity of commonly used methods such as dental chart review, billing data, or practitioner self-report compared with a 'gold standard' of information derived from direct observation of routine dental visits.
METHODS: A team of trained dental hygienists directly observed 3751 patient visits in 120 dental practices and recorded the behaviors and procedures performed by dentists and hygienists during patient contact time. Following each visit, charts and billing records were reviewed for the performed and billed procedures. Dental providers characterized their frequency of preventive service delivery through self-administered surveys. We standardized the observation and abstraction methods to obtain optimal measures from each of the multiple data sources. Multi-rater kappa coefficients were computed to monitor standardization, while sensitivity, specificity, and kappa coefficients were calculated to compare the various data sources with direct observation.
RESULTS: Chart audits were more sensitive than billing data for all observed procedures and demonstrated higher agreement with directly observed data. Chart and billing records were not sensitive for several prevention-related tasks (oral cancer screening and oral hygiene instruction). Provider self-reports of preventive behaviors were always over-estimated compared with direct observation. Inter-method reliability kappa coefficients for 13 procedures ranged from 0.197 to 0.952.
CONCLUSIONS: These concordance findings suggest that strengths and weaknesses of data collection sources should be considered when investigating delivery of dental services especially when using practitioner survey data. Future investigations can more fully rely on charted information rather than billing data and provider self-report for most dental procedures, but nonbillable procedures and most counseling interactions will not be captured with routine charting and billing practices.