Omnidirectional Vision for an Autonomous Helicopter - University of ...

Omnidirectional Vision for an Autonomous Helicopter
Stefan Hrabar and Gaurav S. Sukhatme
Robotic Embedded Systems Laboratory
Center for Robotics and Embedded Systems
Department of Computer Science
University of Southern California
Los Angeles, California, USA
shrabar@robotics.usc.edu, gaurav@robotics.usc.edu
Abstract
We present the design and implementation of an omnidirectional
vision system used for sideways-looking sensing
on an autonomous helicopter. To demonstrate the capabilities
of the system, a visual servoing task was designed
which required the helicopter to locate and move towards
the centroid of a number of visual targets. Results are presented
showing that the task was successfully completed
by a Pioneer ground robot equipped with the same omnidirectional
vision system, and preliminary test flight results
show that the system can generate appropriate control
commands for the helicopter.
1 Introduction
The maneuverability of helicopters makes them particularly
useful aerial vehicles for a number of tasks, especially
where takeoff and landing space is limited, or where
steady flight at low speed is needed. The size and high
cost of operating helicopters do however limit their use for
such tasks. Smaller scale autonomous helicopters could offer
a viable alternative to their full scale versions for use
in many tasks, such as aerial surveillance and communications
bridging.
In order for such an unmanned aerial vehicle (UAV) to
navigate safely through an environment with tall obstacles,
or to monitor features that are in the same horizontal plane
as itself, the UAV must be equipped with sideways-looking
sensors of some sort. Laser range finders provide accurate
range information and have been used for obstacle avoidance
and 3D scene reconstruction [1, 2]. Commercially
available laser altimeters are however limited to scanning
in 2D, are prohibitively heavy and power hungry. Vision
has been used successfully in many feature tracking [3] and
navigation tasks, and is particularly useful because of the
richness in information that is received. Since CCD cameras
are passive sensors, they typically draw little current,
and are light and compactly packaged. Standard lenses,
however, have a limited field of view, so in order to sense a
large area, the camera needs to be actuated. This gives only
partial information about the the scene at any point in time,
and so scene stitching techniques are needed.
An omnidirectional lens can be used to give a 360 degree
semi-spherical field of view [4], enabling the vision system
to track features in the scene without actuating the camera.
Also, multiple features in different parts of the scene
can be tracked simultaneously [5]. The properties of an
omnidirectional lens are well suited to the movement characteristics
of a helicopter. A helicopter can (essentially)
instantaneously move in any direction in 3D space, and the
lens allows it to ”see” in the direction it is moving, without
having to pan/tilt the camera in that direction first.
Omnidirectional vision has been used in many visionbased
tasks [6] such as feature tracking [7], surveillance
[8], navigation [9], 3D scene reconstruction [10, 11],
visual servoing [7], and localization [12]. Vision-based
control of an autonomous helicopter has been done using
a downwards looking camera with a standard lens[13,
14, 15], and omnidirectional cameras have been used on
ground based robots [7, 9] but very little or no work on
vision-based control of a helicopter using an omnidirectional
lens has been done before.
A UAV with omnidirectional sensing capabilities could
be used for various applications. One class of tasks that
it could be used for is locating the centroid of a number
of features in 3D space, and maintaining position at this
centroid. For example, to simultaneously monitor a number
of features, the optimal position for monitoring is likely to
be at their centroid. If the UAV is acting as a hub, this could
be a good position to facilitate communications.

2 Omnidirectional Vision System
Our image capture system is comprised of a NetVision360
omnidirectional lens by Remote Reality, coupled
to a Sony XC-55 black and white CCD camera. The lens
has a field of view of 360 degrees in the horizontal, and
from 35 to 92.5 degrees in the vertical plane. This gives a
semi-spherical field of view, from the horizon down to 35
degrees (if the camera is mounted with the lens pointing
downwards). Since the system was designed for sideways
looking sensing, the blind spot from 0-35 degrees was considered
acceptable. A wireless video transmitter is used to
transmit the video signal to a PC equipped with a frame
grabber, which captures frames at 640x480 resolution.
This introduced transmission noise into the signal, which
hampered the feature detection. In the future, the Linux
stack will be upgraded to handle the vision code, and running
this onboard will eliminate the transmission noise.
Initially the CCD camera was mounted rigidly to the
landing gear of the helicopter, but it was found that the
high-frequency vibrations from the gas engine caused motion
blur in the image. To reduce the vibrations transmitted
to the camera, a mount was designed that decoupled the
camera from the landing gear.
2.1 Helicopter Platform
Our experimental test-bed AVATAR (Autonomous Vehicle
Aerial Tracking And Reconnaissance) [16] is a gaspowered
radio-controlled model helicopter built from a
Bergen chassis. The helicopter carries onboard processing
power and sensors in the form of two PC104 stacks
(one running Linux and one running QNX), a Novatel RT-
2 DGPS board, a compass, an Crossbow VGX IMU, a laser
altimeter and a CCD camera with the omnidirectional lens.
The Linux stack consists of a Pentium II 233MHz processor
and an Imagination PX610a frame grabber. This stack
will is used for image capturing and running vision code.
The QNX stack includes a Pentium III 700MHz processor
and is used for running the low-level helicopter control
code. All the drivers for the sensors are run on this stack,
and the input from the sensors is broadcast to a ground
station for logging. The ground station is a laptop running
QNX. Both stacks are equipped with 802.11b wireless
ethernet cards for communication with the ground station.
This is used to send high-level control commands and differential
GPS corrections to the helicopter. A description
of the AVATAR control architecture can be found in [13].
Figure 2: Omnicam Mount
The mount shown in Figure 2 consists of a frame and a
series of elastic bands that suspend the camera. The elastic
bands were chosen to have a low spring constant such
that the low amplitude, high frequency vibrations of the helicopter
could not be transmitted to the camera. This was
found to greatly reduce the amount of motion blur in the
image as can be seen in Figure 3 (b) compared to Figure 3
(a).
(a) Rigid Mount
(b) Decoupled Mount
Figure 3: Images From Rigid and Decoupled Mounts
showing Blur Reduction
2.2 Pioneer Platform
Figure 1: Helicopter Test-bed
The onboard processing power of the Linux stack was
insufficient for running the vision code, so a 1.8GHhz wireless
video transmitter was used to transmit the video signal
to a second ground station (a 2.4GHz Pentium IV based PC.
For the ground-based experiment, an Active Media Pioneer
2DX was used. The Pioneer was fitted with the omnidirectional
camera by mounting the camera at the end
of a vertical pod 90cm above ground level (as shown in
Figure 4). The camera was mounted with the lens facing
downwards, as it would be on the helicopter. This resulted
in a small portion of the FOV being occluded by the mount,
which affected the Pioneer’s performance in completing the
task (there was no such problem with mount on the helicopter
however). The Pioneer is equipped with a Pentium

II 200 MHz processor, which was used to run the low-level
control code. The code was run using the Player/Stage environment
[17, 18]. As with the helicopter, a 1.8GHz wireless
video transmitter was used to transmit the video signal
to a ground station where the vision code was run. This
code produced high-level control commands at a rate of
8Hz which were transmitted to the Pioneer via the 802.11b
wireless ethernet.
Figure 5: Original Image
Figure 6: Final Image
Figure 4: Pioneer fitted with the Omnicam
2.3 Centroid Finding Task
In order to illustrate the capabilities of the omnidirectional
vision system, a simple task was designed. A number
of visual targets are placed in the environment surrounding
a robot, and the robot is required to locate and move to the
centroid of the targets (a point equidistant from all targets).
The task is simplified to two dimensions by placing all the
targets on the same horizontal plane.
3 Image Processing
3.1 Preprocessing
Before the feature recognition algorithm is run on the
image, various image processing operations are applied to
it in an attempt to highlight the salient features. A 3x3
Median-filter is applied to remove white additive noise [19]
and preserve the edge sharpness [20]. Since much of the
original image is black, the image is negated, reducing the
number of regions it contains. Thresholding is applied to
convert the image from greyscale to binary, and the threshold
value is chosen manually at run-time such that the desired
features are preserved while many undesired features
are lost. The image is then unwarped as described in Section
3.2, followed by segmentation and connected component
labelling as described in [13]. Figure 5 shows the original
spherical image, and Figure 6 shows the image once
the above processing has been performed.
3.2 Unwarping
Our feature detection algorithm (described in Section
3.3) is not invariant to image skew, thus it was nec-
essary to unwarp the spherical image obtained from the
omnidirectional lens. If the geometric properties of such
a lens are known, a projection function can be calculated
for performing the unwarping [21]. Since these properties
were not known for our lens, the unwarping was performed
by mapping points on the spherical image (P x,y ) to corresponding
points in the perspective image (P u,v ) using the
geometrical properties of both images. (see Figure 7). This
geometrical mapping was calculated for each quadrant of
the spherical image. As an example, the mapping for the
first quadrant was:
x = 320 + r*sin(θ)
y = 240 - r*cos(θ)
UnwarpMap[u + v*UNWARP_WIDTH].x = x
UnwarpMap[u + v*UNWARP_WIDTH].y = y
where:
v goes from 0 to UNWARP_HEIGHT
u goes from 0 to UNWARP_WIDTH/4
θ = (u - UNWARP_WIDTH/4)*0.00416611
r = 240 - 0.0000028v 3 - 0.0027v 2 + 1.1v - 0.42
360
0
θ
r
Spherical Image
(640x480)
P(x,y)
UNWARP_WIDTH
P(u,v)
360
UNWARP_HEIGHT
Unwarped Image
(1754x290)
Figure 7: Illustration of the Unwarping Process

3.3 Feature Recognition Using Hu’s Moments
As discussed in [22], Hu’s moments can be used for feature
recognition. Once the system has been trained on a certain
feature, this feature can be recognized regardless of its
orientation, scaling or translation. Although this technique
is not invariant to skewing, it was found that the feature
(the letter H) could still be detected when it was skewed
to 45 degrees horizontally, and 30 degrees vertically. This
technique was previously used successfully for detecting
the same feature painted on a helipad during autonomous
vision based landing [13].
result is that the robot moves towards the centroid of targets.
Omax
Robot
Centroid
Omin
Percieved area of target
Average area of targets
Omax
Omin
4 Centroid Finding Algorithm
Step 1
Step 2
In order to find the centroid of the visual targets, it is
necessary to know the relative distance to each target. Since
the targets were made to be the same size, the area of each
target in the image is used as an indication of the distance
to the target. The algorithm runs as follows:
For each image:
find the area and position of each detected target (x position
only, since the task is in 2D.
(A 1, A 2, A 3, ...A n; X 1, X 2, X 3, ...X n)
Calculate the average, maximum and minimum areas
of the targets and the positions of largest and smallest
targets:(A ave, A max, A min), (X max, X min).
while(A max - A min ≥ threshold)
Calculate the offsets from the average area to the minimum and
the maximum areas:
O max = A max - A ave
O min = A ave - A min
determine the action to take:
if O max ≥ O min
move away from X max
else
move towards X min
loop
This causes the robot to move away from the closest target
or move towards the farthest target, until all the targets
are roughly at the same distance (within threshold). A useful
property of this technique is that it is not necessary to
know the actual distance to each target, only the relative
distances.
Figure 8 illustrates this algorithm. Note that the size
of the solid-line circles represents the relative size of the
targets as seen by the robot, while the dotted-line circles
represent the average size of the targets. In Step 1, O min is
greater than O max , so the robot moves towards the farthest
target. By Step 2, O max becomes larger than O min , and
the robot moves away from the closest target. The overall
Figure 8: Centroid Finding Algorithm
5 Experimental Setup
Although the omnidirectional vision system had been
developed for use on an autonomous helicopter, it was decided
that the system should first be tested on a ground
robot. This would help to eliminate many problems before
using the system on the helicopter, as it is easier to
conduct experiments and debug the code on a ground vehicle
than a helicopter. Experiments were thus first done
on the Pioneer robot, and once the robot could complete
the centroid finding task consistently, the experiment was
conducted on the helicopter. Although autonomous flight
was not yet possible with our new helicopter platform, data
could be collected from all the onboard sensors. The helicopter
was flown under pilot control in the vicinity of the
visual targets while the centroid finding algorithm was run.
The GPS coordinates and heading of the helicopter were
logged as well as the commands that were being generated
by the algorithm. After the test flights the data were analyzed
to see if appropriate control commands were being
generated by the algorithm (the GPS coordinate of the centroid
had been recorded, and the helicopter’s heading and
location were compared to this).
For the Pioneer-based experiment, a 23x17cm white H
on a black background was used as the visual target. This
version of the experiment was designed to be a scaleddown
version of the outdoor one - since the targets would
be much closer to the Pioneer, their size was reduced accordingly
. The scale of the outdoor experiment (i.e. the
maximum distance that the targets could be from the helicopter
for effective recognition) was determined by the
resolution of the camera and size that the outdoor targets
could be made (83x70cm). For the outdoor experiment, the
targets were placed on the circumference of a circle with a
diameter of 12.2m. For the indoor experiment, the diameter
was reduced to 2.5m.

350
300
250
Amax
Area (pixels)
200
150
100
Aave
50
Amin
0
0 20 40 60 80 100 120 140 160 180
Time (s)
Figure 9: Helicopter Experiment being Conducted
Figure 11: Ave, Max and Min target areas over time
that had been correctly detected. In a total of 25 frames,
48% of the H’s were detected, and no false detections were
made.
6.2 Helicopter
Figure 10: Pioneer Experiment being Conducted
6 Experimental Results
6.1 Pioneer
The experiment using the Pioneer robot was conducted
by placing the robot in a position where it could see the
visual targets, but not at the centroid of the targets. The algorithm
was then run as the robot attempted to locate and
move towards the centroid of the targets. The task was considered
completed when the robot had stabilized within 20
cm of the true centroid of the targets. 20 such runs were
completed. The areas of the detected targets were recorded
at 1s intervals, as well as the time for completing each task.
Since the experiment was conducted indoors, GPS could
not be used to determine the position of the robot relative
to the centroid. As an alternative indication of the effectiveness
of the algorithm, the Average, Minimum and Maximum
areas are plotted against time. Figure 11, is an example
of such a plot and shows that the Max and Min values
converge towards the Ave value, indicating that the robot
converges to a point approximately equidistant from all the
targets.
The effectiveness of the feature recognition algorithm
was also evaluated by examining a number of the raw and
processed frames captured, and noting the number of H’s
Four test flights were completed as described in Section
5. The data were analyzed by looking for instances
where at least 2 H’s had been detected, noting the helicopter’s
position and heading at that time, as well as the
control command that was being generated. If the command
was trying to move the helicopter towards the centroid
of the targets, it was considered to be correct. The
analysis showed that appropriate commands were generated
90% of the time.
Figure 12 shows a spherical image taken during flight
as well as the corresponding processed image. The lines
drawn from the center of the processed image to the H’s
indicates that the H’s have been identified. The vertical
line shows the desired heading for the control command
that has been generated for this image.
As for the ground-based experiment, the effectiveness of
the feature recognition technique was evaluated. In a total
of 25 frames, 38% of the H’s were detected, and 8 false
detections were made.
7 Conclusion and Future Work
We have presented the design and implementation of a
sideways-looking sensor for an autonomous helicopter using
an omnidirectional vision system. The effectiveness of
the system was demonstrated by completing a visual servoing
task that required a number of features to be identified
simultaneously within a large field of view in the environment.
Data from several experiments show that the task
was completed successfully on a ground based robot. It

Figure 12: Images Taken During Flight
was shown that the vision system and centroid finding algorithm
could generate appropriate control commands when
tested on a pilot-controlled model helicopter. In the future
we plan to run the full experiment on the helicopter when
it is flying autonomously.
Once we have achieved omnidirectional vision-based
control of the helicopter we intend to use this capability
for tasks such as flying between obstacles (such as buildings)
and visually servoing into position to monitor an area
of interest (such as a specific window on a building).
Acknowledgments
This work is sponsored in part by NASA under
JPL/Caltech contract 1231521 and by DARPA grants
DABT63-99-1-0015 and 5-39509-A (via UPenn) under the
MARS program. We thank Doug Wilson for support with
flight trials.
References
[1] Larry Matthies Charles Bergh, Brett Kennedy and Andrew
Johnson, “A compact, low power two-axis scanning laser
rangefinder for mobile robots,” September 2000.
[2] Ryan Miller and Omead Amidi, “3-d site mapping with the
cmu autonomous helicopter,” in In Proceedings of 5th International
Conference on Intelligent Autonomous Systmes,
June 1998.
[3] Gregory D. Hager and Peter N. Belhumeur, “Efficient region
tracking with parametric models of geometry and illumination,”
IEEE Transactions on Pattern Analysis and Machine
Intelligence, vol. 20, no. 10, pp. 1025–1039, 1998.
[4] S. Baker and S. Nayar, “A theory of catadioptric image formation,”
in Proceedings of ICCV, Bombay, India, January
1998, pp. 35–42.
[5] Terry Boult, “Frame-rate omnidirectional surveillance
tracking, http://www.cse.lehigh.edu/ tboult/track/,” .
[6] Y. Yagi, “Omnidirectional sensing and its applications,”
in IEICE Transactions On Information And Systems, vol.
E82D, no. 3, pp. 568-579, Mar.1999.
[7] Peng Chang and Martial Hebert, “Omni-directional visual
servoing for human-robot interaction,” in Proceedings of
the 1998 IEEE/RSJ International Conference on Intelligent
Robots and Systems (IROS ’98), October 1998, vol. 3, pp.
1801 – 1807.
[8] T. Kanade, R. Collins, A. Lipton, P. Burt, and L. Wixson,
“Advances in cooperative multi-sensor video surveillance,”
in In Proceedings of DARPA Image Understanding Workshop,
volume 1, pages 3–24, November 1998.
[9] N. Winters, J. Gaspar, G. Lacey, and J. Santos-Victor,
“Omni-directional vision for robot navigation,” in Proc.
IEEE Workshop on Omnidirectional Vision - Omnivis00,
2000.
[10] S.B. Kang and R. Szeliski, “3-d scene data recovery using
omnidirectional multibaseline stereo,” in IEEE Conference
on Computer Vision and Pattern Recogstruction, 1996, pp.
364–370.
[11] Roland Bunschoten and Ben Krse, “3-d scene reconstruction
from cylindrical panoramic images,” .
[12] Wasinee Rungsarityotin and Thad Starner, “Finding location
using omnidirectional video on a wearable computing
platform,” in ISWC, 2000, pp. 61–68.
[13] S. Saripalli, J. F. Montgomery, and G. S. Sukhatme, “Visionbased
autonomous landing of an unmanned aerial vehicle,”
in IEEE International Conference on Robotics and Automation,
Washington D.C., May 2002, pp. 2799–2804.
[14] C. Sharp, O. Shakernia, and S. Sastry, “A vision system
for landing an unmanned aerial vehicle,” in In Proceedings
of IEEE International Conference on Robotics and Automation,
2001, pp. 1720-1728.
[15] Pedro J.Garcia-Padro, Gaurav S.Sukhatme, and
J.F.Montgomery, “Towards vision-based safe landing
for an autonomous helicopter,” Robotics and Autonomous
Systems, 2000, (accepted, to appear).
[16] USC Autonomous Flying Vehicle Homepage, “http://wwwrobotics.usc.edu/˜avatar,”
.
[17] B.P. Gerkey K. Sty R.T. Vaughan A.Howard G.S. Sukhatme
and M.J. MAtaric, “Most valuable player: A robot device
server for distributed control,” in IEEE/RSJ Intl. Conf.
On Intelligent Robots and Systems (IROS), Wailea, Hawaii,
2001.
[18] R.T. Vaughan, “Stage: A multiple robot simulator,” in Tech.
Rep. IRIS-00-393, Institute for Robotics and Intelligent Systems,University
of Southern California, 2000.
[19] R.Gonzalez and R.Woods, Digital Image Processing,
Addison-Wesley, 1992.
[20] Ioannis Pitas, Digital Image Processing Algorithms,
Prentice-Hall, 1993.
[21] J. Gaspar, N. Winters, and J. Santos-Victor, “Vision-based
navigation and environmental representations with an omnidirectional
camera,” in In IEEE Transactions on Robotics
and Automation, 16(6):890–898, 2000.
[22] M.K.Hu, “Visual pattern recognition by moment invariants,”
in IRE Transactions on Information Theory, 1962,
vol. IT-8, pp. 179–187.