Desk Monitoring Guide - National Reporting System

a critical element of educational gain measures because programs cannot determine NRS educationallevel advancement or completion without a posttest. Other factors that affect valid assessmentinclude using equated alternate forms of the same test for pre- and posttesting (or using acomputer-adapted test that ensures students get minimal non-overlapping items) and training staff ontest administration.As a state monitor, you want to know whether programs are following the state assessmentpolicies, especially in regard to the number and percentage of students that are pre- and posttestedand the time between the two assessments. Programs that pre- and posttest high percentages of theirstudents at the appropriate times will have more accurate and valid measures of educational gainacross the levels. The following quantitative measures reflecting these assessment processes areamenable to a desk monitoring approach.• Number and percentage of students pretested by a set time. Knowing the proportionof students pretested within a certain time (e.g., at intake, two weeks after enrollment, bycontact hours) allows you to ensure that programs are pretesting all students in a timelyway. A percentage that is too low may indicate a problem with assessment procedures,and data quality will suffer.• Number and percentage pre- and posttested. If the program is not posttesting asufficient number of students, educational gain will be too low. A poor posttestingpercentage also may indicate problems with student retention.• Average contact hours between pre- and posttest. Publishers of standardizedassessments provide guidelines for the minimum number of instructional hours beforeposttesting can detect gains. State NRS assessment policy should establish the time,based on these guidelines, when programs are to posttest. A measure of average contacthours between pre- and posttesting allows a desk monitor to determine whether programsare adhering to this time.A desk monitoring tool can include these measures by level, separately for ABE and ESLstudents, or aggregated for the program overall. If programs vary in the assessments they use, it isalso be advisable to have separate measures for each assessment.A desk monitoring approach can also track whether programs follow the other NRSrequirements for valid assessments (e.g., using a state NRS–approved assessment, using alternateforms or items in the pre- posttesting process, using scores to place students within NRS levels), butit is better to do so through quality control procedures within the program. For example, the NRSdata system could have error checks to verify the use of correct assessments and alternate forms andautomatically assign students into levels on the basis of test scores. Problems in these areas requireimmediate correction and would take too long to identify and correct through desk monitoring.Training local staff in the proper administration of assessment, also a critical part of valid assessmentprocedures, is a professional development issue, which we will discuss later.Goal setting and orientation. The NRS follow-up measures of entered and retainedemployment, entry into postsecondary education or training, and receipt of a secondary credential areoften the most difficult measures for local programs because collecting them involves severalcomplicated and interrelated procedures. Programs must have a good orientation and goal-settingprocess to ensure that appropriate follow-up goals are set, and they need procedures for trackingstudents. Programs that use the survey method for collecting follow-up measures must also have10 Desk Monitoring: Improving Program Performance