data consistency, completeness and cleaning - The INCLEN Trust

DATA CONSISTENCY, COMPLETENESSAND CLEANINGByB.K. Tyagi and P.Philip SamuelCRME, Madurai

DATA QUALITY(DATA CONSISTENCY, COMPLETENESS )High-quality data needs to pass a set of quality criteria. Those include:Accuracy:An aggregated value over the criteria of integrity, consistency, and densityIntegrity:An aggregated value over the criteria of completeness and validityCompleteness:Achieved by correcting data containing anomaliesValidity:Approximated by the amount of data satisfying integrity constraintsConsistency:Concerns contradictions and syntactical anomaliesUniformity:Directly related to irregularities and in compliance with the set 'unit of measure'Density:The quotient of missing values in the data and the number of total values oughtto be known

DATA CLEANSINGData auditing: The data is audited with the use of statistical methods to detectanomalies and contradictions. This eventually gives an indication of thecharacteristics of the anomalies and their locations.Workflow specification: The detection and removal of anomalies is performed by asequence of operations on the data known as the workflow. It is specified after theprocess of auditing the data and is crucial in achieving the end product of highqualitydata. In order to achieve a proper workflow, the causes of the anomalies anderrors in the data have to be closely considered.Workflow execution: In this stage, the workflow is executed after its specification iscomplete and its correctness is verified. The implementation of the workflow shouldbe efficient, even on large sets of data, which inevitably poses a trade-off becausethe execution of a data-cleansing operation can be computationally expensive.Post-processing and controlling: After executing the cleansing workflow, the resultsare inspected to verify correctness. Data that could not be corrected duringexecution of the workflow is manually corrected, if possible. The result is a new cyclein the data-cleansing process where the data is audited again

DATA QUALITYData quality is not linear and hasmany dimensions like Accuracy,Completeness, Consistency, Timeliness andAuditability. Having data quality on onedimension is as good as 'no quality‘.None of the Data Quality dimensionsis complete by itself, and many a timesdimensions are overlapping.

DATA ACCURACYThe address of customer in the customerdatabase is the real address.The temperature recorded in thethermometer is the real temperature.The bank balance in the customer'saccount is the real value customer deservesfrom the Bank.

DATA COMPLETENESSData Completeness definition is the 'expectedcompleteness'. It is possible that data is notavailable, but it is still considered completed, as itmeets the expectations of the user. Every datarequirement has 'mandatory' and 'optional'aspects.For exampleCustomer's mailing address is mandatoryand it is available and because customer’soffice address is optional, it is OK if it is notavailable.

DATA CONSISTENCYConsistency of Data means that data across the enterprise should be in synch with eachother.Examples of data in-consistency are:An agent is inactive, but he still has his disbursement account active.A credit card is cancelled, and inactive, but the card billing status shows 'due'.Data can be accurate (i.e., it will represent what happened in real world), but stillinconsistent.An Airline promotion campaign closure date is Jan 31, and there is a passengerticket booked under the campaign on Feb. 2.Data is inconsistent, when it is in synch in the narrow domain of an organization, but notin synch across the organization.For example:Collection management system has the Cheque status as 'cleared', but in theaccounting system, the money is not shown being credited to the bank account.Reason for this kind of inconsistency is that system interfaces are synchronizedduring the end-of-day batch runs.Data can be complete, but inconsistentData for all the packets dispatched from NEW DELHI to CHENNAI are available.,but some of the packages are also shown as 'under bar-coding' status.

DATA TIMELINESS'Data delayed' is 'Data Denied'The timeliness of data is extremely important. This isreflected in:Companies are required to publish their quarterlyresults with in a given frame of time.Customers service providing up-to date information to thecustomers.Credit system checking on the credit card accountactivity.The timeliness depends on user expectation. An onlineavailability of data could be required for room allocationsystem in Hospitality, but an overnight data is fine for abilling system.

DATA AUDITABILITYData Auditability means that any transaction,report, accounting entry, bank statement etc.can be tracked to its originating transaction.This would need a common identifier, whichshould stay with a transaction as it undergoesTransformation, aggregation and reporting.

DATA CLEANSING• Data cleansing, data cleaning, or data scrubbing is the process of detecting andcorrecting (or removing) corrupt or inaccurate records from a record set, table,or database. Used mainly in databases, the term refers to identifyingincomplete, incorrect, inaccurate, irrelevant, etc. parts of the data and thenreplacing, modifying, or deleting this dirty data.• After cleansing, a data set will be consistent with other similar data sets in thesystem. The inconsistencies detected or removed may have been originallycaused by user entry errors, by corruption in transmission or storage, or bydifferent data dictionary definitions of similar entities in different stores.• Data cleansing differs from data validation in that validation almost invariablymeans data is rejected from the system at entry and is performed at entry time,rather than on batches of data.• The actual process of data cleansing may involve removing typographical errorsor validating and correcting values against a known list of entities. Thevalidation may be strict (such as rejecting any address that does not have a validpostal code) or fuzzy (such as correcting records that partially match existing,known records).

Data Cleaning is the First Step in DataProcessing• Data cleaning is the process of detecting andcorrecting (or removing) incomplete,incorrect, inaccurate and irrelevant parts of adataset by replacing, modifying or deletingthe bad data• It is the first and most important step in anydata processing• It aims to have access to reliable data to avoidfalse and misdirected conclusions

Data Descriptive Document• A document should be developed alongside the raw data containingthe following information:– Variable name - Variable type - Missing values– Variable description - Variable value

Using Excel for Character Data• Select the variable of interest, for example gender• From the main tool bar go to data, from there selectFilter and then “autofilter”• Click on the auto-filter arrows and a box will show allthe available values of our variable• Check the variable values in the data descriptiondocument to determine the valid values• Use auto-filter to select the questionable values• Excel can give you the case ID of each questionablevalue.• Refer to the case ID, check and correct thequestionable value by going back to the medical record

Another Approach:Using Frequencies

Checking for Invalid CharacterValues….(1)• Run frequencies on all character variables that represent alimited number of categories such as gender, residence,hospital’s department, occupation, etc.GENDERFrequency2 1F 300M 440X 1f 3Missing values 5

Checking for Invalid Character Values….(2)• Three categories do not fit with our data valueGENDERFrequency2 1F 300M 440X 1f 3Missing values 5

Checking for Invalid Character Values….(3)• The 2 and the X are inappropriate values.• f depending on the situation, it could be consideredan error or notGENDER2 1Occur onceF 300M 440X 1Occur oncef 3Missing values 5Frequency

Correcting Invalid Character Values• If the lower case values were entered into thefile by mistake but the value, aside from thecase, was correct, we consider this valuecorrect and change each of these lower casevalues to upper case• For the 2 and X values, we need to identifythe location of these errors and correct it afterchecking the medical records

Checking Missing Data• Check each of the cases with missing data(here on gender)• See whether there is information in the casethat allows that variable to be entered (e.g.the patient’s name will generally indicategender)

Checking for Invalid Numeric Values• The techniques for checking invalid numeric data are quite differentfrom the techniques used with character data– Examine minimum and maximum values for each numeric variable– Internal consistency methods; if we see that most of the data values fallwithin a certain range of values, then any values that fall far enough outsidethe range may be data errors– Run a univariate analysis, focusing especially on• Number of non-missing observations, number of observation not equal to zero andthe number of observation greater than zero are of most interest at this stage• Extremes shows the five lowest and five highest values for numeric variables• Quantiles• Mean• Standard deviation to decide on constitute reasonable cutoffs for low and high datavalue• Range• Graphic displays: a stem-and leaf plot, a box plot and a normal probability plot• Check the medical records for the extreme values and write a note to thedata center about the findings to help in further cleaning of these data

Dates: Hospitalization…..(1)• We can create a variablefrom subtracting thedate of discharge fromdate of admission, andcall it totalhospitalization 1• This variable will detectany wrong data entryfor dates such as casenumber 6014

Dates: Hospitalization…..(2)• We can create a variablefrom adding the dayspatient spent in ICU,ward and private roomand call it totalhospitalization 2

Dates: Hospitalization…..(3)• To check inconsistency wecan create a variable, lets callit difference by subtractingthe total hospitalization 1(created from subtractingdates of admission anddischarge) and the totalhospitalization 2 (created bysumming the days spent inICU, ward and private room)• We need to check any valueother than zero by using theauto-filter command andrecheck the medical records