Master Thesis - Fachbereich Informatik

112 CHAPTER 5. RESULTS AND EVALUATION
v[m/min] Ωtotal ΩPTM σtube GT Dmin GT Dmax GT D RMSE
10 1 11.4 0.05 -0.12 0.14 0.01 0.07
20 1 6.9 0.04 -0.16 0.11 -0.02 0.07
30 1 4.6 0.05 -0.19 0.19 0.0 0.07
40 1 3.2 0.07 -0.21 0.17 -0.01 0.09
55 1 2.3 0.07 -0.16 0.16 0.01 0.08
Table 5.4: Evaluation results at different conveyor velocities v for black tubes (50mm length,
∅8mm). The accuracy of the measurements does not decrease significantly with faster velocities
nor with a decreasing number of per tube measurements ΩPTM indicated by the RMSE.
σtube is the per tube standard deviation and GT D stands for ground truth distance (see
Section 5.1.2).
amount of data, more comprehensive representations will be used based on the proposed
evaluation criteria.
Black Tubes The results of the velocity experiments with black tubes are summarized
in Table 5.4.
TheblacktubesshowadetectionrateΩtotal of 1 for all velocities, i.e. no tube has
passed the measuring area without being measured independent of how fast the tubes are
moved. The average number of per tube measurements ΩPTM decreases from 11.4 atthe
slowest velocity (10m/min) to 3.2 at the maximum possible production velocity. Even at
55m/min each tube is measured at least twice. The average standard deviation σtube of
themeasurementspertubereachesfrom0.04 to 0.07mm, again there is only a very little
rise from the slower to the faster velocities. The absolute ground truth distance does not
exceed 0.21 and measurements that are shorter or larger than the ground truth are equally
distributed indicated by the mean ground truth distance GT D that is approximately zero.
As an example, the ground truth distance at 30m/min is shown in Figure 5.8(a). If the
distance is larger than 0, the manually measured length is shorter than the vision-based
measurement and vice versa. Due to the variance in the ground truth data it is not very
likely that the distance is zero for all values. However, the distance should be as small as
possible. If the ground truth distance is one-sided, i.e. all measurements of the system
are larger or shorter than the corresponding ground truth measurement, this indicates an
imprecise calibration factor. The conversion of the pixel length into a real world length
results in a systematical error which has to be compensated by adapting the calibration
factor.
The RMSE differs only marginally between the tested velocities. The largest RMSE
is computed at 40m/min with 0.09. This value is only slightly larger than the deviation of
human measurements. For lower velocities it is even better with 0.07. Another indicator
of how the vision-based measurements converge to the ground truth data is the Gaussian
distribution over the sequence of all measurements. This distribution is based on the
mean µseq and standard deviation σseq (see Section 5.1.2). Figure 5.8(b) compares the
vision-based distribution (solid line) at 30m/min and the corresponding ground truth
distribution (dashed line). The mean is 49.66 in both cases. σseq is slightly larger with
0.1193 compared to the ground truth with 0.1027.
In terms of accuracy and precision this means the vision-based measurements of black
tubes are equally accurate compared to human measurements (laboratory conditions) and