Back Room Front Room 2

This paper describes the data sampling
techniques developed as a basis for the data analysis
component of a data quality software system based
on formal methods (Ranito et al., 1998 and Neves,
Oliveira et al. 1999).
2 FUZZY CLUSTERING FOR
SAMPLING
Several methods can be used to approach sampling
in databases (Olken, 1993) but, in particular,
weighted and stratified sampling algorithms appear
to produce best results on data quality auditing.
Whenever processes are concerned with small
amounts of data exhibiting similar behaviour, the
exceptions and problems emerge in a faster way.
Fuzzy clustering is an interesting technique to
produce weights and partitions for the sampling
algorithms. The creation of partitions is not a static
and disjoint process. Records have a chance to
belong to more than one partition and this will
reduce the sampling potential error, since it is
possible to select a relevant record 1 during the
sampling of subsequent partitions, even when it was
not selected in the partition that shared more similar
values with it. The same probabilities can also be
used to produce universal weighted samples.
The Partition Around Method (Kaufman and
Rousseeuw, 1990) is a popular partition algorithm
where the k-partitions method is used to choose the
centred (representative) element of each partition,
whereupon the partitions’ limits are established by
neighbourhood affinity. The fuzziness introduced in
the algorithm is related with the dependency
between probability of inclusion in a partition and
the Euclidean distance between elements, not only
regarding the nearest neighbour but also other
partitions’ representatives.
2.1 K-partitions Method
For a given population, each record is fully
characterized wherever it is possible to know every
value in all p attributes. Let xit represent the value of
record i in attribute t, 1 � t � p. The Euclidean
distance d(i,j) 2 between two records, i and j, is given
by:
d �
2
( i,
j)
� ( xi1
� x j1)
�...
� ( xip
x jp
1 Critical record in terms of data quality.
2 d(i,j) = d(j,i)
RELATIONAL SAMPLING FOR DATA QUALITY AUDITING AND DECISION SUPPORT 83
)
2
If necessary, some (or all of the) attributes must
be normalized to avoid that different domains affect
with more preponderance the cluster definition. The
common treatment is the calculation of the mean
value of an attribute and its standard deviation (or
mean absolute deviation as an alternative).
The k-partitions method defines as first
partition’s representative the element that minimizes
the sum of all the Euclidean distances to all the
elements in the population. The other representatives
are selected according to the following steps:
1. Consider an element i not yet selected as a
partition’s representative.
2. Consider element j not yet selected and denote
by Dj its distance to the nearest representative of
a partition, already selected. As mentioned
above, d(j,i) denotes its distance to element i.
3. If i is closer to j than its closest representative,
then j will contribute for the possible selection i
as a representative. The contribute of j for the
selection of i is expressed by the following gain
function:
C ji
j
� max( D � d(
j,
i),
0)
4. The potential of selection of individual i as
representative is then given by:
� j
C ji
5. Element i will be selected as representative if it
maximizes the potential of selection:
max
i �
j
2.2 Fuzzy Clustering Probabilities
After defining the partitions‘ representatives, it is
possible to set a probability of inclusion of each
element in each one of the k partitions established,
based on the Euclidean distance between elements.
A representativeness factor fr is set according to
the relevance of an element in the context of each
cluster. Empirical tests indicate this factor to be 0.9
when dealing with a partition’s representative and
0.7 for all the others (Cortes, 2002). The algorithm’s
definition is described below 3 :
3 For a full description see (Cortes, 2002).
C
ji