Download report - ETH - ASL Student Projects

Semester-Project
Corner extraction in
Omnidirectional Images
Autumn Term 2010
Autonomous Systems Lab
Prof. Roland Siegwart
Supervised by: Author:
Stephan Weiss Abraham Oyoqui A.
Laurent Kneip
Markus Achtelik

Contents
Abstract iii
Symbols v
1 Introduction 1
1.1 Related Research . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.2 Wide-angle Lens Model . . . . . . . . . . . . . . . . . . . . . . . . . 3
2 Method 7
2.1 Calibration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
2.2 FAST on a Sphere . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
2.3 First Results and Further Improvement . . . . . . . . . . . . . . . . 9
3 Results 15
3.1 Comparison with the Original FAST . . . . . . . . . . . . . . . . . . 15
3.2 Affine Transformations . . . . . . . . . . . . . . . . . . . . . . . . . . 15
3.3 Application to a Real Image . . . . . . . . . . . . . . . . . . . . . . . 15
4 Conclusions 24
A Comparison using 150 ◦ angle lens 27
B Comparison using 90 ◦ angle lens 29
Bibliography 31
i

Abstract
Omnidirectional cameras present a number of advantages that make them attractive
for robotics applications, e.g. they are cheap and they can represent a large
portion of the environment with a single snapshot which reduces computational
cost. However, omnidirectional cameras introduce radial distortion where lines that
do not pass through the center of the image are strongly bent. This distortion
along with the change of resolution within the same image leads to a deterioration
on the performance of feature detectors when such effects are not modeled. The
purpose of this project is to modify the FAST corner detector, a high-speed and
reliable detector which is suitable for real-time applications, so that it takes into
account the lens model and can accommodate the distortion on omnidirectional
images. Specifically, we aim at applying the FAST algorithm on a spherical model
of the image, the construction of which will implicitly contain the distortion due to
the lens model.
iii

Symbols
Symbols
H Second-order moments matrix for Harris corner detector
C Corner response function
P Linear projection matrix
X Real-world coordinates
λ Depth scale
q Image vector on image sensor plane
u” Real coordinates on image
u’ Pixel coordinates on image
A Rotation matrix for affine transformation
t Translation matrix for affine transformation
r Radius of circle of analysis
rc
Radius of circle of analysis at the center of the image (3)
k Design constant
ρ Distance from center of image. Equivalent to |u’|
f Lens function. Equivalent to g
g Lens model function for projection on omnidirectional image.
In this report a Taylor model.
M Number of pixels analyzed by the FAST.
Can be less than the actual number of pixels on the image.
Acronyms and Abbreviations
FAST Feature from Accelerated Segment Test
RD Radial Distortion
SIFT Shift Invariant Feature Transform
v

Chapter 1
Introduction
The aim of this project is to modify an existing corner detector so that its performance
on omnidirectional images is improved. Specifically, we will use the FAST
(Features from Accelerated Segment Test) corner detector which is a very fast corner
detector and performs well with images taken from cameras that closely follow
the ideal rectilinear pinhole camera model. However on omnidirectional images its
performance decays, mainly due to the change in resolution as we move away from
the center of the image and the radial distortion.
A simple example is depicted in Figure 1.1. A simple pattern will change its shape
and size dramatically even if the changes in perspective are not that great. Figure
1.2 depicts the performance of the original FAST corner detector where only one
out of the 4 corners is detected. At the end of this report we will compare this same
image with the same parameters against the modified FAST detector.
The next section will discuss briefly the existing literature on the subject of feature
detection on omnidirectional images. Afterwards, a brief summary of the theoretical
background concerning the methodology and the general context of this work will
be given.
1.1 Related Research
Corners are a very important cue for vision systems in robotics. Features like
corners or edges are interesting to vision systems because they reduce the amount
of information that has to be dealt with and also a descriptor of the scene can be
made based on these features. The most common approach to finding a corner
or an edge consists of calculating a corner response function on an image. If this
response exceeds a certain value (threshold) then it is considered a corner. Further
discrimination can be made via a non-maximal suppression step to avoid multiple
detections on the same spot.
For example, the Harris corner detector calculates locally the second order moments
of the intensity variations in the vertical and horizontal directions.
⎡
H = ⎣ ( ∂f
∂x )2 ∂f ∂f
∂x ∂y
∂f ∂f
∂x ∂y ( ∂f
∂y )2
From this matrix it has been proven that the largest eigenvalues will determine the
directions of largest change in intensity. Ideally, an edge is an abrupt change in
intensity between two uniform regions in one direction while a corner is a change
in 2 directions. Therefore, a pixel is considered a corner when both eigenvalues on
1
⎤
⎦

Chapter 1. Introduction 2
Figure 1.1: Depending on its position away from the center of the image, a corner
gets different representations
the matrix H are big. However, to avoid calculating the eigenvalues, the following
response function is used
C = |H| − k(traceH) 2
A more extensive revision on the literature on corner detection can be found in
[1]. The corner detector used in this project however does not compute a corner
response function. Instead of defining mathematically an edge or a corner, in FAST
a corner is searched for by looking at the intensities around each candidate pixel.
This means that around each pixel a discretized circle is analyzed, usually of radius
3 and it is considered a corner if there is a continuous sequence of points that are
brighter (or darker) than the analyzed pixel plus (minus) a certain threshold. In
Figure 1.3 we can see an example of this. In this case the threshold was set so
that only 9 points on the circle are found to be brighter than the pixel p. Under
the so-called fast10 (where the test for being a corner requires a sequence of 10
points) this would not be a corner. However, under the fast9 it would be. This will
be important later on where we use the most strict test (fast12 ) to compare the
algorithms.
Changes from viewpoint can be accounted for with these state-of-the-art methods
where the features’ appearance is made invariant to affine transformations, e.g.
changes in scale and rotation. When all of these approaches to corner detection, or
in general to image key-points, are applied to images with radial distortion (RD)
they do not perform correctly. In some research studies like Burschka et al.[2]
it is decided to ignore the effects of RD while in others an image rectification is
performed prior to the application of their feature detection method. Hansen et

3 1.2. Wide-angle Lens Model
Figure 1.2: Corners detected (green) by the original FAST algorithm on an omnidirectional
image
Figure 1.3: FAST criteria for finding corners. Image from Machine learning for
high-speed corner detection. E. Rosten, T. Drummond (2006)
al. uses a spherical representation of the image which is constructed by knowing
the camera parameters. This paper is a modification to the SIFT (Shift Invariant
Feature Transform) in which they “consider defining the scale-space response for
wide-angle images as the convolution of the image mapped to the sphere and the
solution of the (heat) diffusion equation on the sphere”[3]. In this project, a similar
approach was taken in the sense that a spherical model of the images is also built
based on a known camera calibration.
1.2 Wide-angle Lens Model
As it was mentioned, omnidirectional cameras can represent a large portion of the
environment as opposed to perspective cameras which we are used to working with
where a linear mapping is described by a projection matrix P, also called pin-hole
model. However they have in common the fact that both possess a single effective
viewpoint i.e. there is a point through which all rays pass, called the projection
center. This is desirable since it means that any point in the image corresponds to

Chapter 1. Introduction 4
only one direction in the real-world and geometrically correct perspective images
can be generated [4]. There exists a number of different wide-angle lens models
to approximate the actual lens and/or mirror that the camera uses. While some
are approximate function models like the fish-eye transform (FET), polynomial
fish-eye transform (PFET) or division model for example, others are derived projection
functions e.g. equidistant, equisolid, orthographic projection function [5].
The advantage of the first type of models is that they can include errors in the
manufacturing of the lens.
Figure 1.4: An omnidirectional image can capture a whole room in a single snapshot.
Appearance of objects however, change as we move away from the center of the
image.
In this project the Matlab Toolbox from Davide Scaramuzza for camera calibration
was used. As described in the documentation that accompanies the toolbox [6] the
lens model is approximated by a polynomial, the order of which is usually 4. The
coefficients of this polynomial are the calibration parameters to be extracted with
the toolbox.
According to Micusik [7], any pixel in the image point can be represented by a unit
vector in 3D, i.e. on a sphere. The projection equation can then be written as
λq = P · X
with q, a unit vector representing the image point. When using the spherical model
we can assume that there exists always a vector p” with the same direction as q
which is mapped to u” on the sensor plane (Fig 1.5). More formally, there exist two
functions g and h that map from the vector u” on the sensor plane to the vector p”,
one function accounting for the shape of the lens or mirrors and the other for the
type of projection onto the sensor plane. For perspective cameras and orthographic
projection h = 1. Instead of using two different functions, orthographic projection
is assumed and therefore only one function must be calculated.
h(|u”|)u” u”
λq = λ
=
g(|u”|) g(|u”|)
Furthermore we can account for the misalignment between camera and sensor planes
by the following affine transformation:

5 1.2. Wide-angle Lens Model
Figure 1.5: Mapping onto the sensor plane. Image from Micusik.
Figure 1.6: Mapping to sensor plane for hyperbolic mirror and fish-eye lens. Image
from Micusik.
u” = Au’ + t
where u’ is in pixel coordinates. As mentioned the proposed form for the lens
function is polynomial, where the coefficients a0, a1,...,aN have to be calculated
and N is considered 4 for this project as it was found to yield a complex and
accurate enough polynomial.
g(|u”|) = a0 + a1|u”| + a2|u”| 2 + ... + aN|u”| N
Thus the final projection equation for omnidirectional cameras becomes:
λp” = λ
= P · X
u”
a0 + a1|u”| + a2|u”| 2 + ... + aN|u”| N

Chapter 1. Introduction 6

Chapter 2
Method
In this section the method followed for developing the algorithm will be discussed.
Three main parts formed the development of the final version of the modified FAST
algorithm. First, the calibration using Scaramuzza’s Toolbox is used where the
lens model is obtained. Then, again using functions from this toolbox, a spherical
model is obtained. The second part describes the application of the FAST idea to
the spherical model and finally, the adjustments and further modifications to get a
better algorithm.
2.1 Calibration
As illustrated in Figure 2.1 the calibration is done by taking several pictures from
different views of a pattern, usually a checkerboard. Extrinsic parameters as well
as intrinsic are determined. We are interested in the intrinsic, i.e. the coefficients
of the n-order polynomial. We can also obtain the true center of the image which
is helpful later on when we deal with the change in resolution.
Figure 2.1: Calibration of an omnidirectional image. Taking various images of a
checkerboard pattern we can obtain the lens model for the camera as an nth order
polynomial. Image on the right is a “physical” representation of the lens assuming
orthographic projection from the lens to the image sensor plane.
As mentioned in the previous section, an omnidirectional image can be represented
on a spherical model. This comes from the fact that cameras with a single viewpoint
map a pixel on the image to a unique direction in the real world and therefore, to
7

Chapter 2. Method 8
a unique position on a sphere. The idea here is that the circle of analysis for each
pixel on the image for the FAST will now be applied on the 3D representation of
such pixel. In other words, instead of sliding a circle across the image, it will slide
on a sphere where straight lines in the real-world have not yet been bent due to the
projection onto the planar image.
2.2 FAST on a Sphere
Once we have the spherical model of the image we can calculate a circle around
each pixel on 3D. This circle can be thought of as the intersection of the sphere and
a cone starting on the center of the sphere. We know that the sphere is unitary so
the only parameter to be designed is the radius of this circle, or equally the angle
of the cone. To be consistent with the original FAST the radius was chosen to be 3.
While the radius remains constant, its projection to the image will vary depending
on where it is on the sphere. Figure 2.2 depicts this projection of the circles, and it
is easy to see that those circles located near the z axis of the sphere (i.e. the center
of the image) will project like a perfect circle on the image and will therefore be
equivalent to the discretized circle for the original FAST. What is more interesting
for us however is the circles on the regions away from the center of the image where
we will have a change in the shape which will account for the line bending effect on
the outer regions of the image.
Figure 2.2: Projection of circles on the sphere to the image plane
Figure 2.3 and 2.4 show a more detailed representation of this situation. We can
conclude that the distortion model has an effect on the change of shape of these
circles and these changes will be able to deal with the line bending. Note that what
is being projected are not the whole circles but rather only 16 of the points (Fig.
2.5). The FAST algorithm will be applied to these 16 points like it is applied to
those of a discretized circle in the normal FAST. The difference here is that we no
longer have discretized circles but real coordinates which introduces the need for
interpolation in order to avoid aliasing. We chose bilinear interpolation to do this
which means that for every point we need to save 4 coefficients on a look-up table.

9 2.3. First Results and Further Improvement
This results in a 16x4xM matrix, M being the number of pixels in the image.
Figure 2.3: A projected circle near the center of the image maintains its shape,
being equivalent to the original FAST
Figure 2.4: A projected circle away from the center of the image will become an
ellipse. It is this change in shape that implicitly takes into account the distortion
model.
2.3 First Results and Further Improvement
As said, the FAST algorithm is applied on these 16 points i.e. the intensity values
at each of these points is taken, interpolating between its 4 closest pixel coordinates.
These intensity values undergo the FAST criteria looking for a sequence of n
connected points that are brighter or darker than the pixel around which the circle
was built (plus/minus a threshold).
Figure 2.6 shows the first comparison between the original FAST and the FAST on
a sphere. From now on, all the images presented are taken with the most strict test
for the FAST criteria: fast12. This was done to ensure that when using the FAST
on the sphere, corners can only be detected if they actually are the result of two
straight lines forming a corner. If it passes under the fast12 it passes under every
other version. In general, we can find more corners with the FAST on the sphere
but a closer look at the found corners reveals that interpolation is playing a big
part in finding these corners and not necessarily the modification through the lens
model.
As we can see on Figure 2.7 the shape of both versions of FAST is similar. However
one of them fails to recognize the corner while the other will be able to satisfy the
conditions to determine a corner since it is interpolating values. In other words,

Chapter 2. Method 10
Figure 2.5: A circle is projected as an ellipse and only 16 points are taken for the
FAST algorithm. Interpolation is required for each of these points.
Figure 2.6: Left: original FAST. Right: FAST on a sphere
intensity values get pulled by neighbouring pixels and in corners such as the one
depicted, interpolation will actually aid the detection. While desirable, this is not
what is looked for since we have not yet seen the effect of the implicitly modeled
distortion function on the detection of corners.
Figure 2.8 shows the result when the algorithm is applied on a pattern more suitable
for evaluation of corner detection. As we can observe, performance of the modified
FAST is similar to the original version. Specifically we have the same detection
issues in the outer regions of the image where the modified algorithm should outperform
the original one. A closer look to the sphere hinted that the method should
be detecting the corners. However, when projecting the circle to the image it is clear
that the circle is too small for the resolution present in that region. We have not
yet taken into account the fact that resolution varies nonlinearly as we move away
from the center of the image.
As a first approximation though, it was considered that resolution varied linearly
and so the radius of the circle would grow linearly too as we go away from the
center. The proportion between the distance from the center of the image and the

11 2.3. First Results and Further Improvement
Figure 2.7: Effect of interpolation. Points in green are the points considered for the
original FAST while points in red are the projected points from the circle on the
sphere.
radius of the circle is then tuned to attain detection at the borders.
r = rc + kρ
However, to avoid this tuning and take into account the nonlinear variation of
resolution, the lens model was again taken into account for calculating the increase
of the circle radius.
r = rc + k∆f
r = radius of circle at distance ρ from the image center
rc = radius of circle at the center of the image (3)
k = design constant. Note: It could be changed but leaving it to 1 gives good results
∆f = f(ρ + ∆ρ) − f(ρ)
∆ρ = 3
Figure 2.10 shows a lens model function for the 190 ◦ camera used for all images so
far. We basically use the slope of the function to determine how big the radius of
the circle should be at a certain distance from the center. When evaluating this
algorithm we can see that we finally get corner detection at the edge of the image
without having to tune the radius of the circle.

Chapter 2. Method 12
Figure 2.8: Comparison between original FAST and FAST on a sphere with circle
of constant radius. Top: original FAST. Bottom: FAST on sphere

13 2.3. First Results and Further Improvement
Figure 2.9: Modified FAST with a linearly increasing radius of the circle of analysis.
Top: k = 4 Bottom: k = 6
Figure 2.10: Lens function for a 190 ◦ camera

Chapter 2. Method 14
Figure 2.11: Modified FAST with nonlinearly increasing radius of the circle of
analysis

Chapter 3
Results
Now that we have derived the modifications for the FAST algorithm we can evaluate
its performance. The evaluation presented in this section is qualitative. In order
to fully conclude on the performance of the FAST on a sphere with respect to the
original version, it is necessary to devise a method to quantitatively measure their
performance.
3.1 Comparison with the Original FAST
We will now show some comparisons between the original FAST and the proposed
modification. Figure 3.1 and 3.2 are representative images of the experiments performed
with the camera positioned in such a way that affine transformations are
avoided or they are minimal. In such circumstances, we get the same detections as
the original FAST on the central regions of the image and more detections as we
move away from the center. This indicates that the modified algorithm is in fact
outperforming the original version. Even on those perspectives where both algorithms
fail to recognize corners that clearly are present we can see on the borders,
on parts of the image that are not the pattern, how the FAST on a sphere is able
to detect corners better than the original algorithm.
3.2 Affine Transformations
We have already seen in Figure 3.2 that under some orientation of the corners with
respect to the camera, they will not be detected and the original FAST and the
FAST on a sphere will perform almost the same. We will now see another failure
mode. When we introduce an affine transformation the FAST on a sphere will cease
to detect also on the edges of the image. Even if we see that the modified FAST
detects a couple of corners more than the original it is not conclusive enough to say
that one performs better than the other. In fact, the edges detected are mainly due
to the interpolation effect rather than to the lens distortion function.
3.3 Application to a Real Image
As a final result we will show the performance on a highly distorted image featuring
a more realistic setting. While the pattern used showed us some of the potential
modes in which the algorithm fails, it is also desirable to test it on a more complex
image. Further testing with these kind of images can be made to statistically
measure the repeatability of the algorithm.
15

Chapter 3. Results 16
As we said before, all testing with the patterns were done with the fast12, the most
strict criteria for corner discrimination and a low threshold to eliminate the possibilty
of not detecting a corner by idealizing too much the change in intensities on
an edge or corner. Now that we have seen that the modified version outperforms
the original one, especially on the edges, we can tune the FAST, relax the criteria
in order to find the parameters that will give a good performance for a given application.
In the case of Figures 3.6 and 3.7 these parameters were chosen to be a
threshold of 50 and relax the FAST test to only 9 points, or fast9.
In the same way, we can compare now the image that was presented at the beginning
as motivation. We can again see that with the same parameters, the modified FAST
performs better than the original, finding all corners of the rectangle.
Even though this is a qualitative comparison it can already be seen that the modified
algorithm is capable in general to detect many more corners than the original FAST.
It must still be evaluated if this improvement is worth the added computational cost
and processing time.

17 3.3. Application to a Real Image
Figure 3.1: Comparison between original and modified FAST. fast12, threshold =
25. Top: original FAST. Bottom: FAST on sphere

Chapter 3. Results 18
Figure 3.2: Comparison between original and modified FAST. fast12, threshold =
25. Top: original FAST. Bottom: FAST on sphere

19 3.3. Application to a Real Image
Figure 3.3: Comparison between original and modified FAST in a setting where
there exists an affine transformation between camera and pattern. fast12, threshold
= 25. Top: original FAST. Bottom: FAST on sphere

Chapter 3. Results 20
Figure 3.4: Comparison between original and modified FAST in a setting where
there exists an affine transformation between camera and pattern. fast12, threshold
= 25. Top: original FAST. Bottom: FAST on sphere

21 3.3. Application to a Real Image
Figure 3.5: Comparison between original and modified FAST in a setting where
there exists an affine transformation between camera and pattern. fast12, threshold
= 25. Top: original FAST. Bottom: FAST on sphere

Chapter 3. Results 22
Figure 3.6: original FAST
Figure 3.7: Modified FAST with nonlinearly increasing radius of circle. fast9,
threshold = 50

23 3.3. Application to a Real Image
Figure 3.8: original FAST
Figure 3.9: Modified FAST with nonlinearly increasing radius of circle. fast9,
threshold = 50

Chapter 4
Conclusions
The proposed modifications to the FAST corner detector oriented to improving its
performance on omnidirectional images have been evaluated qualitatively and have
been found to successfully increase correct detection of corners. In general, it has
been found that it will detect at least about the same amount of corners as the
original implementation and when avoiding affine transformations, it will detect a
lot more corners along the edge of the image. While satisfactory results, a more
quantitative measure is still needed. Specifically we would like to compare the
amount of correct detections between both implementations and the repeatability
of the modified FAST.
The fact that under affine transformations and in some orientations the corner
detector fails already gives an indication that repeatability might not be so high
and the detector by itself would not be able to find the same corner on a different
image. However, we saw that the FAST can be tuned to achieve good performance
by increasing the threshold while relaxing the number of points constraint or vice
versa. Therefore if we would know or have an approximation of the relative motion
of the camera between images we could adjust these parameters and have a higher
probability of finding the same corner again. Interpolation also becomes an issue
when encountered with artificial lines i.e. areas that under a certain perspective
appear so thin they become a line. Together with discretization errors these lines
are detected as corners. Since the normal FAST also suffers from discretization
errors, interpolation will in general help to get a better detection as long as we
remove artificial lines. If this is not possible we could also process further the
corners detector and ensure there are not many points along the same line.
Furthermore, this modifications only make sense for lenses with high distortion such
as the one presented in this report. In appendix A some images are included for
lenses with 150 ◦ and 90 ◦ lens angle. Detection is basically the same for the original
and modified versions since lines are not bent so hard and resolution changes are
also small. Although performance is not increased, it gives good signs that the
modifications did not affect the normal performance of the detector.
Finally, speed must still be evaluated. The very strength of the FAST corner detector
is its speed that make it useful for real-time applications. The implementation
of this algorithm requires the creation of very large look-up tables for the position of
the points of the circle around each pixel and the coefficients necessary for interpolation.
With each look-up table containing a matrix of 2x16xM for the circle points
coordinates and 4x16xM for the interpolation coefficients, if we use a very large
resolution this already becomes a problem regarding memory available. M is the
number of pixels that will be examined. This value can also be reduced if we eliminate
the black regions around the image but typically for an image of 480x640, M
is around 300,000. Hence the need to evaluate if resolution can be reduced enough
24

25
to allow such a look-up table to be stored in the vehicle’s memory and evaluate if
it can process images in real-time.

Chapter 4. Conclusions 26

Appendix A
Comparison using 150 ◦ angle
lens
Figure A.1: Original FAST with a variable radius on a 150 ◦ lens
27

Appendix A. Comparison using 150 ◦ angle lens 28
Figure A.2: Modified FAST with a variable radius on a 150 ◦ lens

Appendix B
Comparison using 90 ◦ angle
lens
Figure B.1: Original FAST with a variable radius on a 90 ◦ lens
29

Appendix B. Comparison using 90 ◦ angle lens 30
Figure B.2: Modified FAST with a variable radius on a 90 ◦ lens

Bibliography
[1] M. Lourenco, J. Barreto, A. Malti: Feature Detection and Matching
in Images with Radial Distortion. In 2010 IEEE International Conference on
Robotics and Automation, Anchoragee, Alaska, USA 2010.
[2] D. Burshka, M. Li, R. Taylor, G. Hager: Scale-invariant registration
of monocular endoscopic images to ct-scans for sinus rugery. In MICCAI (2),
2004
[3] P. Hansen, P. Corke, W. Boles, K. Daniilidis: Scale-Invariant
Features on the Sphere. In International Conference on Computer Vision, Oct.
2007
[4] D. Scaramuzza: Omnidirectional Vision: From Calibration to Robot Motion
Estimation. PhD Thesis, Diss. ETH No. 17635. 2008
[5] C. Hughes, P. Denny, E. Jones, M. Glavin: Accuracy of fish-eye lens
model. Applied Optics/ Vol. 49 No. 17, 2010
[6] D. Scaramuzza: Autonomous System Lab. OCamCalib Toolbox. Internet:
http://robotics.ethz.ch/~scaramuzza/Davide_Scaramuzza_files/
Research/OcamCalib_Tutorial.htm [Aug. 2010]
[7] B. Micusik: Two View Geometry of Omnidirectional Cameras. PhD thesis,
Center for Machine Perception, Czech Technical University in Prague, 2004.
[8] D. Schneider, E. Schwalbe, H. -G. Maas: Validation of geometric models
for fisheye lenses. ISPRS Journal of Photogrammetry and Remote Sensing
64 (2009) 259-266
[9] E. Rosten, T. Drummond: Machine learning for high-speed corner detection.
In European Conference on Computer Vision, 2006
31