BACKGROUND: The Victorian Audit of Surgical Mortality (VASM) seeks to peer review all deaths associated with surgical care in Victoria, Australia. The effectiveness of the VASM as an educational and quality improvement tool is dependent on the accuracy of source data it receives. We aimed to examine the accuracy and quality of source data provided by the treating surgeon for peer review, and the inter-rater concordance level between the external validator findings and the treating surgeon. METHODS: Of the 629 cases that completed the VASM audit second-line peer review process over a 4-year period (from 1 July 2012 to 30 June 2016), a total of 32 (5%) were randomly selected for the external validation process. The blinded external validator was impartial to the VASM audit, and was provided only de-identified patient medical records. The analysis for the checked and validated data points and their concordance was determined using Gwet's agreement coefficient, which provides a stable inter-rater reliability coefficient not affected by prevalence and marginal probability. RESULTS: The inter-rater concordance analysis suggested that there is a high level of agreement (82.9% overall) between the treating surgeon and external validator. The use of thromboembolism deterrent stockings was the only variable where agreement was poor (52.4%) with a Gwet score of 0.10 (-0.40 to 0.60). CONCLUSION: The inter-rater concordance analysis results support the validity of the VASM process, which is dependent on the accuracy of data submitted by the treating surgeon.
BACKGROUND: The Victorian Audit of Surgical Mortality (VASM) seeks to peer review all deaths associated with surgical care in Victoria, Australia. The effectiveness of the VASM as an educational and quality improvement tool is dependent on the accuracy of source data it receives. We aimed to examine the accuracy and quality of source data provided by the treating surgeon for peer review, and the inter-rater concordance level between the external validator findings and the treating surgeon. METHODS: Of the 629 cases that completed the VASM audit second-line peer review process over a 4-year period (from 1 July 2012 to 30 June 2016), a total of 32 (5%) were randomly selected for the external validation process. The blinded external validator was impartial to the VASM audit, and was provided only de-identified patient medical records. The analysis for the checked and validated data points and their concordance was determined using Gwet's agreement coefficient, which provides a stable inter-rater reliability coefficient not affected by prevalence and marginal probability. RESULTS: The inter-rater concordance analysis suggested that there is a high level of agreement (82.9% overall) between the treating surgeon and external validator. The use of thromboembolism deterrent stockings was the only variable where agreement was poor (52.4%) with a Gwet score of 0.10 (-0.40 to 0.60). CONCLUSION: The inter-rater concordance analysis results support the validity of the VASM process, which is dependent on the accuracy of data submitted by the treating surgeon.