Evaluation of IT modernisation in the NHS - NETSCC

Report to SDO for NCRS ProjectReview of Economic Implicationsleast some primary data analysis, and which compared two or more IT systems orone setting before and after introduction of an IT system.Due to the rapidchanges in IT, it was agreed that studies published in the last decade would bestinform the aim of the review.In the process of developing the scope andmethods for this study, it became clear that there is little consensus on thedefinition of large scale IT intervention. Therefore, we used a broad set of termsin order to maximize the sensitivity of our search.We accepted at the outset that we were not aiming to estimate the size of aspecific effect. Studies were likely to be very heterogeneous, e.g. in differentsettings, implementing different IT systems, evaluating implementation againstdiverse outcomes etc., so calculating a pooled answer was, a priori, notappropriate. However, given the anticipated benefits of IT modernisation in theNHS, 54 we wanted to identify the extent to which empirical evidence might supportthese anticipated benefits.A further source of heterogeneity is the different methods used in empiricalstudies. Thus a second objective of our review was to describe the diversity ofmethods used in the empirical evaluation of large scale IT implementation inhealth care. Although we intend to describe the empirical findings of the mostrelevant papers, we were equally interested in the methods that have been used.3.3.1. Inclusion / exclusion criteriaThe abstracts or titles of the complete set of references were reviewed by tworeviewers (EP and AH). The main exclusion criteria at this stage of the reviewwere studies reporting findings on micro-interventions, clinical decisionsupport interventions, internet advice-based interventions, telemedicine,opinion papers and letters. Furthermore, reviewers classified all abstracts onthe basis of the perceived likelihood that papers (a) contained empirical data(empirical: probably empirical: possibly empirical. not empirical) or (b) usedinnovative methodology (methodological; probably methodological; notmethodological). All abstracts classified as being neither empirical normethodological by both reviewers, or “neither” by one reviewer and “possiblyempirical” by the other, were excluded at this stage.26