Master Thesis - Fachbereich Informatik

100 CHAPTER 5. RESULTS AND EVALUATION
truth data has also a certain variance. Thus, the distributions measured by humans and
a machine vision system may differ. This gets critical if two distributions intersect.
Tubes that are actually too short or too long, but are measured to be within the tolerance
are denoted as false positives (FP). On the other hand, tubes of an allowed length can be
wrongly classified as outlier and are denoted as false negatives (FN). More mathematically,
false positives and false negatives can be defined as follows:
FP = {T |T ∈ G ′ 0 ∧ T /∈ G0} (5.3)
FN = {T |T /∈ G ′ 0 ∧ T ∈ G0} (5.4)
In terms of system evaluation, the following measures can be used:
ΩFP = NFP
Ntotal
ΩFN = NFN
Ntotal
(5.5)
(5.6)
where NFP and NFN indicate the number of false positives and false negatives respectively.
Both the false positive ratio ΩFP and the false negative ratio ΩFN should be zero
in the optimal case. As already discussed in the introduction, ΩFP is more critical than
ΩFN, since it is less bad to sort out a good tube than delivering a failure to the customer.
Performance The performance of the system can be evaluated with respect to the average
processing time that is needed to analyze a frame:
ΩTIME = 1
M
M�
i=1
ti
(5.7)
where M is the number of frames considered and ti represents the processing time
of frame i. ΩTIME is expressed in terms of ms/frame. This measure can be used to
determine the maximum possible capture rate. Skipped frames indicate that the camera
captures more frames than the system is able to process.
Qualitative Criteria
Standard Deviation Per Tube The multi-image measuring approach is based on the
idea, that more robust measuring results can be reached if each tube is measured several
times. In the ideal case, all measurements should yield the equal length value. In practice,
however, the single measurements can differ. The standard deviation σtube(i) canbeused
as an indicator of how much these measurements vary. It is computed as:
�
�
�
σtube(i) = � 1
mi � � �2 lj(i) − l(i)
mi − 1
j=1
(5.8)