Use of kappa statistic in determining validity of quality filtering for meta-analysis: a case study of the health effects of electromagnetic radiation.

The objective of this study was to determine the validity and reliability achieved in employing trained graduate students to read and evaluate the methodologic content of papers analyzing the health effects of electromagnetic field radiation. The search identified 11,730 references from multiple sources, of which 8759 were unique and from which 97 were selected for inclusion in a meta-analysis. Without reference to the graduate students' assessments, an epidemiologist read and scored all 97 articles as well. A 10% random sample of the 97 articles was selected for a repeat assessment by the epidemiologist, who did not review her first assessment. Interrater validity was good (kappa > or = 0.45-0.75) for 15 of the 23 criteria. Intrarater reliability was excellent (kappa > 0.75) or good in 21 of the 23 rating criteria. We concluded that valid and reliable results can be obtained by training graduate students to screen papers for methodologic content.