data consistency, completeness and cleaning - The INCLEN Trust

Another Approach:Using Frequencies

Checking for Invalid CharacterValues….(1)• Run frequencies on all character variables that represent alimited number of categories such as gender, residence,hospital’s department, occupation, etc.GENDERFrequency2 1F 300M 440X 1f 3Missing values 5

Checking for Invalid Character Values….(2)• Three categories do not fit with our data valueGENDERFrequency2 1F 300M 440X 1f 3Missing values 5

Checking for Invalid Character Values….(3)• The 2 and the X are inappropriate values.• f depending on the situation, it could be consideredan error or notGENDER2 1Occur onceF 300M 440X 1Occur oncef 3Missing values 5Frequency

Correcting Invalid Character Values• If the lower case values were entered into thefile by mistake but the value, aside from thecase, was correct, we consider this valuecorrect and change each of these lower casevalues to upper case• For the 2 and X values, we need to identifythe location of these errors and correct it afterchecking the medical records

Checking Missing Data• Check each of the cases with missing data(here on gender)• See whether there is information in the casethat allows that variable to be entered (e.g.the patient’s name will generally indicategender)

Checking for Invalid Numeric Values• The techniques for checking invalid numeric data are quite differentfrom the techniques used with character data– Examine minimum and maximum values for each numeric variable– Internal consistency methods; if we see that most of the data values fallwithin a certain range of values, then any values that fall far enough outsidethe range may be data errors– Run a univariate analysis, focusing especially on• Number of non-missing observations, number of observation not equal to zero andthe number of observation greater than zero are of most interest at this stage• Extremes shows the five lowest and five highest values for numeric variables• Quantiles• Mean• Standard deviation to decide on constitute reasonable cutoffs for low and high datavalue• Range• Graphic displays: a stem-and leaf plot, a box plot and a normal probability plot• Check the medical records for the extreme values and write a note to thedata center about the findings to help in further cleaning of these data

Dates: Hospitalization…..(1)• We can create a variablefrom subtracting thedate of discharge fromdate of admission, andcall it totalhospitalization 1• This variable will detectany wrong data entryfor dates such as casenumber 6014

Dates: Hospitalization…..(2)• We can create a variablefrom adding the dayspatient spent in ICU,ward and private roomand call it totalhospitalization 2

Dates: Hospitalization…..(3)• To check inconsistency wecan create a variable, lets callit difference by subtractingthe total hospitalization 1(created from subtractingdates of admission anddischarge) and the totalhospitalization 2 (created bysumming the days spent inICU, ward and private room)• We need to check any valueother than zero by using theauto-filter command andrecheck the medical records

ACKNOWLEDGEMENTWe thank• Prof. Donald and Miss. Yara