Fourth National Incidence Study of Child Abuse and Neglect (NIS–4)

each child substantiated by CPS or thought to meet the study criteria on either type ofdata form and deciding their “countability” in relation to the study definitions. Theyassessed each alleged form of suspected or substantiated maltreatment as to its substance(who did what to whom, when, with what effect, and with what quality of evidence),rating the degree to which each required element met the countability criteria of the HarmStandard and of the Endangerment Standard. Overall assessments of the child’scountability under each definitional standard summarized the ratings across all allegedmaltreatment forms.In the NIS–4, a specialized Computer-Assisted Evaluative Decision System(CAEDS) assisted evaluative coders as they assessed cases. CAEDS offered a number ofadvantages over the paper-processing approach used in previous NIS cycles: it offeredautomated reminders of the definitions and implemented consistency checks as the coderskeyed their decisions; it allowed simultaneous and efficient electronic access to dataforms by primary evaluative coders and reliability coders (as well as the unduplicationteam); it served as the management system for supervisors to allocate coding assignmentsand monitor reliability of individual coders in an ongoing way; and it protectedconfidential data by maintaining the forms electronically on a secure password-protectednetwork, eliminating the need to provide specialized physical security during their useand transfer.The benefits of using CAEDS for ongoing monitoring of the consistency ofcodes and intercoder reliability measures are evident in the reliability of the NIS–4evaluative coding decisions. Despite the complexity of the assessments, results werequite reliable, as gauged by two measurements. The first was the simple percentage rateof agreement between the initial evaluative coder and another, reliability coder whoblind-coded a random 40% of the child records (i.e., 12,334 children). The inter-coderreliability agreement rate was 97% for both Harm Standard countability andEndangerment Standard countability. The second reliability measure was the Kappacoefficient, which takes into account the level of agreement expected by chance, based onthe overall distribution of the codes for the item. The Kappa scores were .95 foragreement on Harm Standard countability and .91 for agreement on EndangermentStandard countability.The NIS–4 Analysis Report (Sedlak, Mettenburg, Winglee et al., 2010) givesdetails of the NIS–4 definitions and the evaluative coding process.2–14