Overt versus covert assessment of observer reliability.

This study examined the tendency of observers to make less reliable recordings of behavioral events when a calibrating observer is absent. Experienced observers in 4 different research sites coded videotapes of family interaction using the multicategory system developed at each site. 60 tapes were coded simultaneously by randomly selected observer pairs (overt reliability assessment) and 40 tapes were coded independently by 2 observers who did not know that their entries could be compared (covert assessment). Within site, intraclass correlations (ICCs) were computed separately for both forms of reliability assessment and a variety of behaviors. Overt ICCs were very high for most behaviors in all 4 systems. The corresponding covert reliabilities were significantly lower. Covert decline was conspicuous in the first 10 min of a 1-hour coding session. Hence, observer fatigue was not its principal cause. Apparently, observers lapse into a less attentive "set" prior to coding without a partner. This tendency is most discernible when a highly complex system is employed.