Rigid Body Reconstruction for Motion Analysis of Giant Honey Bees ...

icg.tugraz.at

Rigid Body Reconstruction for Motion Analysis of Giant Honey Bees ...

Rigid Body Reconstruction for Motion Analysis of Giant Honey Bees using

Stereo Vision

Michael Maurer, Matthias Rüther and Horst Bischof

Institute for Computer Graphics and Vision (ICG), Graz University of Technology

maurer/ruether/bischof@icg.tugraz.at

Gerald Kastberger

Institute for Zoology, Karl-Franzens University Graz

kastberger@uni-graz.at

Abstract

Zoologists are interested in the rapid movements of

giant honeybees. Especially the mass-movement of bees

during the defense behavior is of interest. State of the

art approaches measure the motion of a single bee using

a laser vibrometer, which does not provide geometric

information on mass-motion like speed, intensity,

starting point or mesh density. To overcome this problem,

a vision based measurement system is proposed.

A portable stereo setup using two high resolution cameras

with high frame rates is designed to acquire image

sequences of a defense wave in an outdoor environment.

The functionality of the acquisition setup has

been proven at an expedition to Nepal. Additionally, a

framework to segment bees and reconstruct their motion

is presented. The challenging task of solving the

corresponding problem is solved using reduced graph

cuts, to get accurate matches in the presence of repetitive

patterns. An evaluation of the method has been

done by comparison to manually labeled data.

1. Introduction

Giant honey bees (lat. apis dorsata) belong to the

species of honey bees and live in the mainly forested

areas of southern and southeastern Asia, like Nepal.

The subspecies apis dorsata dorsata that was observed

in this project has the second largest individuals among

all honey bees. Typical workers are about 17 to 20mm

long. A colony of giant honey bees consists of up to

100, 000 individuals, living in an open-nest. The nest

is made up of a single comb with up to two meters in

horizontal span and about one meter in vertical span.

The nests are usually built in exposed places far off the

Figure 1. Giant honey bee colony located

at the backside of a hotel.

ground, like overhanging cliffs, trees or buildings (see

Figure 1 and 2). Adult bees cover the comb in multiple

layers, loosely fixed to the comb. In literature this

construction is addressed as bee curtain, and is parted

in two regions. One shows locomotion of individuals,

while the other one remains stable. In this zone, which

is of special interest to us, the bees mostly remain still

and are uniformly oriented with the heads up.

One method of defense against enemies is the defense

waving, also called shimmering behavior. It is a

mass event, comparable to the Mexican wave of people

during sports events, and propagates over the bee

curtain at a speed of about one meter per second. The

waves are generated by coordinated rising of the abdomen

of neighboring bees (see Figure 3) [1], [5].

From the viewpoint of zoologists a number of geometric

parameters of the nest are of interest. These are:

• the mesh width, i.e. distance between neighboring


Figure 2. Giant honey bee colonies located

at a 20m high water tower.

passive measurement principle. An active observation

would possibly disturb the bees and force a counterattack.

Stereo reconstruction was selected as a suitable

method. An image acquisition system has been constructed

which allows to monitor a complete nest from

one side, at a synchronized frame-rate of 60fps. No

extra illumination is required and the system is autonomously

powered. From a methodological viewpoint,

we faced the challenge of segmenting thousands

of bees per image, establishing correct stereo correspondences

on a very repetitive bee curtain, and tracking

segmented individuals over several hundred frames.

2. Data Acquisition

We constructed a stereo camera setup with the following

restrictions in mind:

• The bee cluster has a size of about 700 × 700 ×

100mm 3 (height × width × depth), that should be

in the measurement range.

Figure 3. Bee cluster performing a wave

by synchronized rising of their abdomen.

The time between the images shown is

83ms.

bees on the curtain,

• the motion of the wave in 3D space,

• the speed in the different directions and

• the participation of second layer bees that are bees

under the surface of the curtain.

Additionally, it is of interest to discriminate between

active bees, which participate in the wave propagation,

and passive bees, which are just dragged along by

the others. Technically, to be able to solve these requirements,

a segmentation of individual bees, a threedimensional

(3D) reconstruction of the bee curtain, and

tracks of individual bees are desired. Having a 3D reconstruction

of the bees over time, their movement can

be evaluated. This evaluation also includes the participation

of second layer bees (below the ‘surface ’layer)

that can only be observed by global movements of the

bee curtain.

The technical challenge hereby lies in the field of

view, defined by the dimensions of the curtain, the high

wave propagation speed, the remote location, and the

• Cameras cannot be mounted arbitrarily close to the

cluster, to avoid unwanted interaction and counterattacks.

A clearance distance of two meters has to

be maintained.

• The size of an individual is approximately

20mm × 6mm (length × diameter). In an image

the diameter should cover at least 10px.

• The reconstruction error in z-direction should not

exceed one millimeter.

• An abdomen flip lasts for about 0.2s and should be

resolved with 10 frames.

• The recorded sequence should contain one complete

wave, so continuous acquisition time must be

greater than one second.

• The stereo setup has to be placed outdoors, possibly

in a high, remote location. It has to be portable,

flexible and withstand influences like dust.

• The setup has to be self powered and should be

able to operate for one working day.

A resolution of 4Mpx per camera was selected. According

to the requirements, 700mm will fit to 1728px

and one pixel will correspond to roughly 0.4mm in the

real world. A bee with a diameter of 6mm will be represented

by about 15px. The cameras are able to capture

60fps and so resolve an abdomen flip with 12 frames.


Two cameras at a frame rate of 60fps result in a data

stream of 480MB/s that has to be stored. The captured

image sequence is buffered to RAM and written to disc

later. This leads to RAM constraining the capture time.

A total of 8GB of RAM results in an acquisition time of

15s. The setup is powered by a 12V PC power supply

and a car battery of 100Ah. So an operation time of

seven hours can be reached. The resulting stereo setup

is shown in Figure 4.

parts of neighboring bees, which poses a challenge for

segmentation.

An evaluation of various segmentation methods, including

MSERs [2], shape prior segmentation [4] and

template matching, led to the conclusion that template

matching is the most efficient and accurate solution to

the problem [3]. Segmentation of individuals was done

by template matching using normalized cross correlation

(NCC). To be robust to variations in size and orientation,

a number of scaled and rotated templates were

matched, using a winner-takes-all strategy. An exemplary

result is shown in Figure 6.

Figure 4. Image acquisition setup consisting

of two cameras mounted to a camera

stand, the industrial PC, a touch screen

and an external power supply.

Figure 6. Segmentation of the single bees

using template matching.

3. Methodology

To be able to analyze the defense waves, individual

bees have to be segmented, matched, tracked and triangulated

based on the stereo images.

Figure 5. Input image.

Having a close look at an input image (see Figure 5),

the bees cover the comb in multiple layers and overlap

closely. Heads and the thoraxes contrast weakly from

the background and the semi-transparent wings cover

Establishing stereo correspondences between segmented

individuals is still challenging, because of the

apparently high similarity among them - otherwise segmentation

by template matching would not have been

successful.

Having a set of bees in discrete positions, and an initial

estimate of disparity by robustly fitting a plane to

the cluster, we formulate the matching process as a discrete

optimization problem, and solve it by using reduced

graph cuts, as proposed by Zureiki et al. [6]. Exemplary

results are shown Figure 7.

In a final step, individuals should be tracked over

time in the stereo images. Figure 8 illustrates the abdomen

flip of a single bee and shows the apparent problems.

These are the rapid flip of the abdomen, motion

blur, and the extreme change in appearance. By tracking

a bee at its thorax, we have to deal with occlusions

caused by the raised abdomen.

Work on tracking is still in progress. We propose

to use an adaptation of the Lukas-Kanade algorithm,

where we seek to track the thorax position from the last

n frames in every new frame. Again, the tracking result

with highest confidence wins. In contrast to the naive

approach of tracking only between consecutive frames,

we are hereby more robust and exploit the symmetry in


224

14 160

231

142 252

302

410

447

84

585 50 14

205

104 143

159 175 227

293 510 518

160

231

142252

302 422 50

216

4 410

557

447

205

196 208

413

110

34 496

104

292303

24

299

330349404

143

159 175 227

293 510 518 422216

241 43

266 267 245 281

542196

208 41

584 587

34 496292303

299

235

173 554 594 24

283

130

158

202

279612

136 149

584 266267 245 281

542

222

418

221 508 553

560

625

130

283

235

173

202 222

54

416

279

398238

554594

612

67579

418

625

45381506

183 109 221 158508

553

499

115 137

150

288

556

602

62

183 109

490

286 16115

80

13253

45381

506499

288

137 556 150602

531

181 228

96 286

289107459

490

16

8 5

62

8

7

45

576

454

531 425

116 148

41

581

181 228 425

256 432

45

576 4. Preliminary Results and Conclusion 96

23 498 176 290

616 548

116

134

441

581 454

256

520

264 374

193 23 498 176 125

290 616

125

171

2

574

9

613 376 567

162

430

10

105 443

46

623 264

In this 374paper a520

non-invasive193

vision system for monitoring

a cluster 623 of giant honey 430bees is presented. For

223 570 171

209

179 291 271 361

551 162

630105

51 480 593

419

179 291 271 361

610 589

613

443

46 51

10

348 170 452 151 174

103

5

187 248

209 419

72

551 630 the first 480time it is

593

possible to analyse the mass behavior

512

151

436120 139 624 603

348

280 300

170

174

562

504

423

378

436120 139 603

452

558 280 122

300

94 504

608

626

85

624of these insects in-situ. The image acquisition system

607

122

237

242

95

47

562 423

was

206 505 232 378 successfully applied during an expedition 558 to Nepal,

294 157

287 138 588 260 229 52206

485 473

79

500333

56

232 294 157

94

528 where a total 505 of 3.6TB

112 106 146 622 138

260229

of image data 52was acquired.

444

287 588

250

106 244

421

333

24

485

500 56

473

198

3

282 546

198

182117

244421

561 282 92546

555 595 596

79 622 Using the proposed reconstruction approach, it is

possible to robustly

470

284

12

559 15615

484 182 segment 117 over

463

561 580 92individuals in an

12

152

470 284

36 243 565 156166

619 484

63 44253

559 area of 500 × 500mm

621 90 36 243 565

2 . The correspondence

166 619 problem

598

under

239 185

128

564

316

1

9621

the presence90

of repetitive patterns is addressed

128

(a) 466 Camera 5011

185

230

320 316 239

9 564

(b) Camera 2

using reduced graph cuts. There, using a stereo image

05 48 55296

320 629

258

265

93

127

pair 629

78 258 327

with 265 618 127 630 and 521 segmented bees respectively, we

327

301 601 494

93

retrieved 436 correspondences,

78

98% of

618

which were correct

577 by manual inspection.

Figure 7. Corresponding577

465

465

1 407 59 583 601 bees resulting

464

153 81 571 479

583 64

from the minimum cut.

464

78 523

59

627 88

153397

81

268 599

615 571

270

415 543

479 64

The tracking

526 397 problem

188 69

495144

353184

277 582578

627

615 is addressed using an adaptive

495 template 144

88

415 543 526

204

259

26 188 69617

427

600

353

387 114

114

75

259

26

617

matching approach, but research is still in

427

262 346 450

progress there. Having the positions of the single bees

118 383 536

60075

614476

469

487387

95

536

in 3D after triangulation,

111

469 367487

it is possible to perform an automated

analysis

525

213

445

111

367

3

597

614

606

476

275 203

525

319 18 213

445 of the z-movement and the mesh width

319

8 433

604 472

in the first layer.

347108

350

18

474

41 35

481

573347

350 Future work will include an exhaustive processing

5

511

83

165

201

41 35609

481 573 511

83

and evaluation of

411

274 486 478

609 the image data gathered in the last

73 580 155 620 155

363

344

190 313

411

620expedition, which sums up to 521 stereo movies. Processing

time for a single movie currently lies at 14hrs,

344 313

740517

516 28437

which will also be improved in the near future.

Figure 8. 28

308 380 8

325 461

161 386

457 Sequence 516of a single bee flipping

the abdomen at a frame rate of 60fps. 308 8

7 312611

332

80

408 337

306 435 532

461

482 586

332

References

180

408 337 532

25

477

310

502468

240

515 477 240

515

the abdomen motion. We further enforce the epipolar

constraint by projecting corresponding tracks in the left

and right image on the mean epipolar line (i.e. mean

y-coordinate in rectified images).

A qualitative result is given in Figure 9, showing the

z-motion of three individual bees over a time period

which covers two consecutive defense waves.

Figure 9. Z-movement of three single bees

tracked over two consecutive defense

waves.

[1] G. Kastberger. Communication and defense behavior

of the asian giant honeybee apis dorsata.

http://www.apis-dorsata.info. last visited: 2010.01.30.

[2] J. Matas, O. Chum, M. Urban, and T. Pajdla. Robust wide

baseline stereo from maximally stable extremal regions.

In In British Machine Vision Conference, volume 1, pages

384–393, 2002.

[3] M. Maurer. Rigid body reconstruction for motion analysis

of giant honeybees using stereo vision. Master’s thesis,

Graz University of Technology, 2010.

[4] M. Werlberger, T. Pock, M. Unger, and H. Bischof. A

variational model for interactive shape prior segmentation

and real-time tracking. In International Conference

on Scale Space and Variational Methods in Computer Vision

(SSVM), Voss, Norway, June 2009. to appear.

[5] Wikipedia. Apis dorsata.

http://en.wikipedia.org/wiki/Giant honey bee. last

visited: 2010.01.30.

[6] A. Zureiki, M. Devy, and R. Chatila. Stereo matching

using reduced-graph cuts. In ICIP07, pages I: 237–240,

2007.

More magazines by this user
Similar magazines