photo-identification software for bowhead whale images

SC/60/BRG24.
PHOTO-IDENTIFICATION SOFTWARE FOR BOWHEAD WHALE
IMAGES
GILBERT R. HILLMAN * , KELLY D. TRASK + , KATHRYN L. SWEENEY # , ANDREW R. DAVIS + , WILLIAM R. KOSKI + ,
JULIE MOCKLIN # AND DAVID J. RUGH #
* 5030 Marathon Drive, Madison, WI, 53705, USA.
+ LGL Limited, environmental research associates, 22 Fisher St., P.O. Box 280, King City, Ontario, L7B 1A6,
Canada.
# National Marine Mammal Laboratory, Alaska Fisheries Science Center, National Marine Fisheries Service,
National Oceanic and Atmospheric Administration, 7600 Sand Point Way, NE, Seattle, WA 98115-0070, USA.
ABSTRACT
A computer program has been developed for identification of individual bowhead whales in aerial photographs. The program
enables the user to create a collection of images and to compare an image to the collection. The collection is ranked by the
program based on similarity to the query image, facilitating identification of the individual in the image. The program has been
tested by the developer (GRH) with 20 images and it appears to compute its matching functions correctly. Eleven successive
versions of the program were provided to LGL Limited, who tested the program with a collection of 78 images and have provided
reports on bugs in the system and feature requests to the developer. All known bugs have been fixed and many features have been
added to make the program perform better. Additional modifications to the program are likely to improve matching performance
and will be identified when larger numbers of images have been processed through the program.
KEY WORDS: BOWHEAD WHALE, PHOTO-ID, MARK-RECAPTURE, ARCTIC, COMPUTER-MATCHING
INTRODUCTION
The photographic database of the Bering-Chukchi-Beaufort Seas (BCBS) population of bowhead whales
(Balaena mysticetus) contains almost 18,000 images collected from the late 1970s to 2007. More than half of the
14,682 images that have been classified according to image quality are of adequate quality to determine if the
whales are marked and could be recognized if they were photographed in the future. All of the images in the
database, but particularly the high quality images, have provided useful information on life-history parameters
including calving intervals (Miller et al., 1992; Rugh et al., 1992a), growth rates (Koski et al., 1992, 1993),
population structure (Davis et al., 1983; Koski et al., 1993, 2006, 2007; Angliss et al., 1995), population size
(Rugh, 1990; da Silva et al., 2000; Schweder, 2003; Schweder and Sadykova, 2007; Koski et al., 2008) and
survival rates (Whitcher et al., 1996; Zeh et al., 2002). These data continue to be used in various on-going
studies to further describe various parameters of bowhead whale life history and population dynamics (Rugh et
al., 2007).
The size of the bowhead photographic database has increased to the point where collection of new
photographs results in a relatively high probability of re-sighting a previously photographed marked whale. The
proportion of the population that has been photographed has increased, and consequently, the likelihood of
photographing a whale that has been photographed during previous studies has increased dramatically. Thus,
new photographs at this time provide much more information both on individual whales and the population than
the same number of photographs did during the early years of bowhead whale photography studies (Schweder
and Sadykova, MS). However, as the number of marked whales in the collection has increased, the time required
to match each new photograph to earlier marked whales has also increased. To assist with the matching of the
rapidly-growing number of photographs in the BCBS photographic collection, we have developed a computerassisted
matching program. This paper describes the development and current capabilities of the program and
discusses future testing and development that will improve the efficiency of the program.
METHODS
Programming
All programming was done using the Matlab (The Mathworks, Inc., Natick, MA) development system. This is
the same system that has been used for earlier matching programs for sea otters and gray whales and has
replaced the Visual Basic/C++ languages used for dolphin matching (Hillman et al., 2003; Markowitz et al.,
2003). The program was written using Matlab’s scripting language and graphical user interface (GUI) developer.
The resulting files can be provided to any potential user who has a current Matlab installation. They can also be
compiled by the programmer to produce MS Windows executable files, which run on any Windows XP or Vista
machine. A Matlab runtime file, which the programmer is permitted by Matlab to distribute to end-users, must
also be installed along with the executable file. For this project, the Matlab runtime and the executable files were
1

SC/60/BRG24.
sent to LGL Limited (LGL) and the National Marine Mammal Laboratory (NMML), and personnel there
successfully installed and ran the program.
Entering images into the program
Bowhead whale images (TIF or JPG files) are opened in the matching program; other common file formats could
be accommodated. When an image is opened, the program automatically creates specific subfolders into which
the program will insert the data about the whale’s marking pattern that will be computed from the user’s input.
The information for each image in a folder will be added to these subfolders by the program as the images are
entered.
Once an image is open, the user selects “Geometric Transform” from a menu. The image appears in a new
window next to a provided “standard” image, which is simply a whale image of excellent quality used as a
reference image to help map features on the new image to be entered into the system. The user clicks on two
points in the new image, such as the tip of the rostrum and the end of the fluke, and on the corresponding two
points in the standard image. The program scales, rotates, and translates the new image so that all images have
the same rotation and scaling. The normalization parameters are saved and are applied to that image each time it
is read by the program.
Some images were not sufficiently normalized by this procedure because the whale was flexed or twisted in the
image. Therefore, an “Un-twist” option was added to the image capture procedure. The user selects “Un-twist”
from the menu and then clicks on the tip of the rostrum, a series of at least four points along the anatomic
midline and the base of the fluke. The program performs a nonlinear distortion of the image that causes the
designated points to fall on the line between the rostrum and the fluke, with interpolation of all pixels between
the selected points. When the control points have been selected, the image is rotated such that the first and last
points fall on the same horizontal row. A spline is then drawn through the intermediate points, with a spline point
computed at each pixel column. Each column is then revolved along its vertical length so that the spline points
all fall on the horizontal between the snout and the tail. The entire image is then rotated back to its original
orientation. This process is fairly slow (at least several seconds), so it should not be used unless there is enough
curvature or twisting to disrupt the location of identifying marks. This process is automatic once the six or more
control points have been selected.
The user then selects the markings on the whale that are to be used for identification. First, a mouse click selects
each mark. If the mark is a small dot, no further action is needed. If it is a large mark or has a distinctive shape,
the user may click on a series of points around the mark to outline it, and the data about its size and shape are
stored for use in matching. Users may review an image and add, edit, or delete stored marks without re-doing the
entire mark selection for that image. This feature allows updating of mark information for a whale from second
and subsequent photos of the same whale. This ensures that the mapped image is an accurate representation of a
whale and permits us to incorporate information from multiple images of a whale into a single image for
matching.
Because some regions of a whale are not clearly visible on some photographs due to wash, glare, etc., the user
may designate all (the program default) or part of each image as a region of interest (ROI) for matching. This is
done by clicking around the ROI to create a polygon that includes only the ROI. If such a region is designated,
the program will disregard marks outside of the ROI in other images during the matching process. If no ROI is
selected, the entire image is treated as the ROI.
The scaling and orientation, un-twisting, ROI, and locations and descriptions of the distinguishing marks are
automatically stored in the subfolders, and are recalled by the program as needed.
To enter, normalize, and mark an image takes 3-7 minutes, depending on the quality of the image and how many
spots must be marked or outlined. When this process is complete, the user selects “Add to Excel File.” This adds
the file path and name of the new image to a list maintained in a MS Excel file; this file stores the names of all
files in the catalogue and the identity of the whale in the image. When the image is first added to the file, the
program adds the identifier “0000” as its ID, later to be changed by the user when the image is identified either
as a new individual or as a match to a previous image. The files listed in the Excel document are the ones to
which each new image is matched.
Matching Images
The user selects any image for identification. In order to be matched to the rest of the collection, the image must
have been previously normalized and its spots marked. If the normalization and spot marking procedures have
not been completed for the selected image, a notice reminds the user to complete them.
2

SC/60/BRG24.
Since, the size, orientation and shape of scars are sometimes distinctive, the user is given an option to indicate
the shape of any mark rather than indicating only its location. For example, this option would be used if a whale
had a distinctively-shaped mark so that all whales with a similar-shaped mark at the same location would be
ranked higher on the list of potential matches than they might be without selection of this option (Fig. 1). The
parameters measured for each mark are: location, area (number of pixels), eccentricity (a numerical indication
of the shape of the spot, ranging from 0 (round) to 1 (straight line)), orientation (angle in degrees from horizontal
of the major axis), length (the major axis, measured in pixels, of an ellipse that approximates the mark) and
perimeter (number of pixels). These parameters are calculated automatically once the user has outlined a spot
with the mouse. If a spot’s location is marked, but the spot is not outlined, the system assumes that the spot is a
small circle. When marks are matched, the difference between them is the Euclidean distance in the data sevendimensional
space of these parameters (x position, y position, area, eccentricity, eccentricity×orientation [i.e., if
it is round its orientation is not relevant], length and perimeter). Each of the parameters is weighted by the
default value unless other values are selected. The default weight is 1 for all parameters except eccentricity, for
which the weight is 10 (because its native value is limited to 0 to 1) and 0.01 for area. The user can set the
weighting value to zero to force a parameter not to contribute to the matching process, or set the weight to any
higher number to have a characteristic of the mark weighted more heavily. The program saves the new weights
in a file called “wtfile.mat.” The capability for the user to vary the weights is a part of the development process,
as experience will tell us which of the computed parameters are most helpful for identification, and therefore
should be weighted more heavily, and which, if any, should be counted less heavily or not at all. It is also
possible that the user may wish to adjust the weights according to the properties of the query image
Any two images may be compared by reading one as “Image 1” and the other as “Image 2.” A “Compare two
images” button measures and displays the difference between the two images. This quick matching function is
useful during testing of the program and for new users to learn how changing options affects the matching
process.
In the usual operation of the program, the user does not open an “Image 2”, but instead, clicks a “Compare to
List” button. In this case, all of the images listed in the Excel file are matched to Image 1. The program measures
the dissimilarity between the query image and each listed image, and displays a list of images ordered from the
lowest to the highest dissimilarity value. As a result, the images that are most similar to the query image are at
the top of the list. It then displays a window with the sorted list and the dissimilarity value. The program also
displays the top six images on the list along with file names and ID’s (Fig. 2). Each of these six images has a
button that allows the image to be displayed in a separate large window, with its selected marks shown. If the
user sees a match among those six images, he enters the ID of the query image in a provided text box. If a match
is not seen, the user may view the next six. The user can scroll through the catalogue of images in either
direction, displaying them six at a time, until a match is found, or the user is convinced that there is no match.
Either a pre-existing or a new ID may be entered, and it is recorded in the Excel file instead of the default
“0000”.
The presence or absence of a match is confirmed by a trained observer with assistance of the program. The goal
of the program is to reduce the number of images that must be reviewed. If the program worked perfectly, its
first suggestion would always be a match, if one existed. This ideal goal cannot always be achieved because
some matching errors are inevitable, some images are unclear and some whales either lack distinctive markings
or have markings resembling those on other individuals. Nevertheless, if a correct match occurs at or near the top
of the list, the amount of labour previously needed to search for matches will be reduced..
Comparison to earlier programs
The overall design of the program is modelled on previous photo-ID programs that were developed for other
species such as dolphins, sea otters, and gray whales (Hillman et al., 2003; Markowitz et al., 2003). The
bowhead program code is entirely new, with many features that are specific to the bowhead whale application.
The processes of file access, Excel file access, image display and mouse-use take advantage of Matlab functions.
The scale-rotation-translation normalization process uses Matlab’s linear conformal transformation. However,
the un-twisting function is a new algorithm. This process is fairly slow (at least several seconds), so it should not
be used unless there is enough curvature or twisting to disrupt the location of identifying marks.
Matching function
The matching function works as follows, with the process repeated as the query image is matched in turn to each
image in the collection:
• The program recalls the locations and shape properties of the two images (A and B) being matched.
3

SC/60/BRG24.
• If A or B has a designated ROI, the marks in the other image are tested to determine whether they fall
within this region. Any marks that are outside the other image’s valid region are temporarily discarded.
• The Euclidian distance is determined between each point in A and B in a 7-dimensional space
comprised of the X distance, Y distance, mark area, mark eccentricity (a scale from 0 to 1 indicating shape from
a circle to a line), angular orientation of the mark, length of the major axis of the mark and perimeter of the
mark, each parameter weighted by its weighting value. The closest points are determined. If the images have
unequal numbers of marks, the smaller number is used (e.g. if A has 3 and B has 5 marks, the closest 3 pairs of
points are found)
• Using a linear translation function and Matlab’s optimization algorithm, the relative position of the two
images is adjusted to minimize the distances between the corresponding closest-fit points.
The difference values between the query and all other images are accumulated and sorted, establishing the order
in which images are presented for inspection.
A second matching algorithm was also implemented after discussion with an engineering colleague, Dr. Hemant
Tagare at Yale University. Rather than the successive-approximations optimization of the point locations, the
algorithm uses an analytical minimization of the summed distance, with a specified number of iterations. The
advantage of this method is that it can weight the distances in such a way that the matching value is weighted
more heavily by points that fit more closely than those that are distant matches. The result of this weighting is to
reduce the influence of outlying marks that might be due to errors or artefacts. The user of the program can select
either of the two matching algorithms; experience will tell whether one is actually more effective than the other.
DISCUSSION
A computer-assisted matching program has been developed using 20 bowhead whale images and has been tested
with a set of 78 images. These relatively small collections were used to develop the program and determine
whether the programming was free of errors, but we have not used the program with larger numbers of images.
A much larger data set is now being prepared for testing. The program runs without bugs or crashes, but does not
yet reliably trap user errors or other unexpected input. The program can be improved by adding features such as
internal help files and additional user error trapping. The latest version of the program was provided to LGL and
NMML on 23 April 2008.
A closer examination of weights is needed to determine relative influence of distance, area, etc. of marks on the
similarity of a matched pair of images; it is still uncertain what values to assign to these weights in order to
provide the most accurate matching. In order to facilitate the matching process, the program also needs to allow
more flexibility in organizing data files. Most importantly, the program needs to be tested using a large
collection of images that have already been matched to determine how far down the list of suggested matches a
correct match may occur. Additional features that may improve the reliability of the matching process have been
requested by the testers at LGL and NMML. Throughout the development process there has been frequent and
effective communication between the programmer and the testing groups, and the result is a working draft of a
program that may be extremely helpful for matching bowhead whale images.
ACKNOWLEDGEMENTS
Development and testing of the computer-assisted matching program was by funded Minerals Management
Service (MMS) through an inter-agency agreement between MMS and NMML. We thank Chuck Monnett of
MMS for administering the contract and comments on earlier drafts of this paper. We also thank Sally Mizroch
and Kim Goetz of NMML and Rolph Davis of LGL for providing comments to improve it.
REFERENCES
Angliss, R.P., Rugh, D.J., Withrow, D.E. and Hobbs, R.C. 1995. Evaluations of aerial photogrammetric length
measurements of the Bering-Chukchi-Beaufort seas stock of bowhead whales (Balaena mysticetus). Rep. int.
Whal. Commn 45:313-24.
da Silva, C.Q., Zeh, J., Madigan, D., Laake, J., Rugh, D., Baraff, L., Koski, W. and Miller, G. 2000. Capturerecapture
estimation of bowhead whale population size using photo-identification data. J. Cetacean Res.
Manage. 2(1):45-61.
4

SC/60/BRG24.
Davis, R.A., Koski, W.R. and Miller, G.W. 1983. Preliminary assessment of the length-frequency distribution
and gross annual recruitment rate of the western arctic bowhead whale as determined with low-level aerial
photogrammetry, with comments on life history. Rep. to Natl Mar. Mammal Lab., Natl Mar. Fish. Serv.,
Seattle, WA by LGL Ltd. Paper SC/35/PS5 presented to the IWC Scientific Committee, June 1983
(unpublished). 91pp.
Hillman, G.R., Wuersig, B., Gailey, G.A., Kehtarnavaz, N, Drobyshevsky, A., Araabi, B.N., Tagare, H.D., and
Weller, D.W. 2003. Computer-assisted photo-identification of individual marine vertebrates: a multi-species
system. Aquatic Mammals 29(1):117-123.
Koski, W.R., Davis, R.A, Miller, G.W., and Withrow, D.E. 1992. Growth rates of bowhead whales as
determined from low-level aerial photogrammetry. Rep. int. Whal. Commn 42:491-499.
Koski, W.R., Davis, R.A., Miller, G.W. and Withrow, D.E. 1993. Reproduction. pp. 239-74. In: J.J. Burns, J.J.
Montague and C.J. Cowles (eds) The Bowhead Whale. Special Publication. No. 2. Society for Marine
Mammalogy, Lawrence, Kansas. 787pp.
Koski, W.R., Rugh, D.J., Punt, A.E. and Zeh, J. 2006. An approach to minimise bias in estimation of the lengthfrequency
distribution of bowhead whales (Balaena mysticetus) from aerial photogrammetric data. J. Cetacean
Res. Manage. 8:45-54.
Koski, W.R., George, J.C. and Zeh, J. 2007. A calf index for monitoring reproductive success in BCB bowhead
whales (Balaena mysticetus). Paper SC/59/BRG7 presented to the Int. Whal. Commn Scientific Committee,
Anchorage, AK, 7-18 May 2007. 6pp.
Koski, W.R., Mocklin, J., Davis, A.R., Zeh, J., Rugh, D.J., George, J.C and Suydam, R. 2008. Estimates of
Bering-Chukchi-Beaufort bowhead whale (Balaena mysticetus) abundance from 2003-2005 photoidentification
data. Paper SC/60/BRG18 presented to the Int. Whal. Commn Scientific Committee, Santiago,
Chile, 1-13 June 2008.
Markowitz, T.M, Harlin, A.D, Wuersig, B. 2003. Digital photography improves efficiency of individual dolphin
identification. Mar. Mamm. Sci. 19(1):217-223.
Miller, G.W., Davis, R.A, Koski, W.R., Crone, M.J., Rugh, D.J., Withrow, D.E. and Fraker, M. 1992. Calving
intervals of bowhead whales -- An analysis of photographic data. Rep. Int. Whal. Commn 42:501-506.
Rugh, D. 1990. Bowhead whales reidentified through aerial photography near Point Barrow, Alaska. Rep. int.
Whal. Commn (Spec. Issue 12):289-94.
Rugh, D.J., Miller, G.W., Withrow, D.E. and Koski, W.R. 1992a. Bowhead whale calving intervals established
through photographic reidentifications. J. Mammal. 73:487-490.
Rugh, D., Koski, W., and George, J. 2007. Interyear re-identification of bowhead whales during their spring
migration past Barrow, Alaska, 1984-2004. Paper SC/59/BRG2 presented to the IWC Scientific Committee,
Anchorage, AK, 7-18 May 2007. 9pp.
Schweder, T. 2003. Abundance estimation from multiple photo surveys: confidence distributions and reduced
likelihoods for bowhead whales off Alaska. Biometrics 59:974-83.
Schweder, T. and Sadykova, D. 2007. Event history models for capture-recapture surveys with passively marked
individuals. Paper SC/59/BRG28 presented to the Int. Whal. Commn Scientific Committee, Anchorage, AK,
May 2007. 14pp.
Schweder, T. and Sadykova, D. MS. Breaking the square root law. (unpublished). 9pp.
Whitcher, B., Zeh, J., Rugh, D., Koski, W. and Miller, G. 1996. Estimation of adult bowhead whale survival
rates. Paper SC/48/AS12 presented to the IWC Scientific Committee, September 1997 (unpublished). 22pp.
Zeh, J., Poole, D., Miller, G., Koski, W., Baraff, L. and Rugh, D. 2002. Survival of bowhead whales, Balaena
mysticetus, estimated from 1981-1998 photoidentification data. Biometrics 58:832-40.
5

SC/60/BRG24.
Fig. 1. The value of using the “Weight Attributes” function is illustrated in the program ranking of the above
images as matches. The top left panel shows whale 1 with spots marked. The top right panel shows whale 2 with
spots marked. The bottom panel shows marks of both whales 1 (plus signs) and whale 2 (circles) superimposed
on whale 2. These two whales are not a match, but without using the Weight Attributes function the program
recognizes these 2 whales as a good match because 5 marked spots match very well. If however, the Weight
Attributes are adjusted to account for the large distinct mark #3, the match probability is decreased and more
accurate matching rank is achieved.
6

SC/60/BRG24.
Fig. 2. Search results screen (front) and main screen (behind).
7