Views
2 years ago

Explorations in Automatic Image Annotation using Textual Features

Explorations in Automatic Image Annotation using Textual Features

Explorations in Automatic Image Annotation using Textual

Explorations in Automatic Image Annotation using Textual FeaturesChee Wee LeongComputer Science & EngineeringUniversity of North Texascheeweeleong@my.unt.eduRada MihalceaComputer Science & EngineeringUniversity of North Texasrada@cs.unt.eduAbstractIn this paper, we report our work onautomatic image annotation by combiningseveral textual features drawn from thetext surrounding the image. Evaluation ofour system is performed on a dataset ofimages and texts collected from the web.We report our findings through comparativeevaluation with two gold standardcollections of manual annotations on thesame dataset.1 IntroductionDespite the usefulness of images in expressingideas, machine understanding of the meaning ofan image remains a daunting task for computers,as the interplay between the different visualcomponents of an image does not conform to anyfixed pattern that allows for formal reasoning ofits semantics. Often, the machine interpretation ofthe concepts present in an image, known as automaticimage annotation, can only be inferred byits accompanying text or co-occurrence informationdrawn from a large corpus of texts and images(Li and Wang, 2008; Barnard and Forsyth,2001). Not surprisingly, humans have the innateability to perform this task reliably, but given alarge database of images, manual annotation isboth labor-intensive and time-consuming.Our work centers around the question : Providedan image with its associated text, can weuse the text to reliably extract keywords that relevantlydescribe the image ? Note that we are notconcerned with the generation of keywords for animage, but rather their extraction from the relatedtext. Our goal eventually is to automate this taskby leveraging on texts which are naturally occurringwith images. In all our experiments, we onlyconsider the use of nouns as annotation keywords.2 Related WorkAlthough automatic image annotation is a populartask in computer vision and image processing,there are only a few efforts that leverage on themultitude of resources available for natural languageprocessing to derive robust linguistic basedimage annotation models. Most of the work hasposed the annotation task as a classification problem,such as (Li and Wang, 2008), where imagesare annotated using semantic labels associated toa semantic class.The most recent work on image annotation usinglinguistic features (Feng and Lapata, 2008)involves implementing an extended version ofthe continuous relevance model that is proposedin (Jeon et al., 2003). The basic idea underlyingtheir work is to perform annotation of a test imageby using keywords shared by similar trainingimages. Evaluation of their system performanceis based on a dataset collected from the news domain(BBC). Unlike them, in this paper, we attemptto perform image annotation on datasetsfrom unrestricted domains. We are also interestedin extending the work pursued in (Deschacht andMoens, 2007), where visualness and salience areproposed as important textual features for discoveringnamed entities present in an image, by extractingother textual features that can further improveexisting image annotation models.3 Data SetsWe use 180 images collected from the Web, frompages that have a single image within a specifiedsize range (width and height of 275 to 1000 pixels).110 images are used for development, whilethe remaining 70 are used for test. We create twodifferent gold standards. The first, termed as Intuitiveannotation standard (GS intuition ), presents auser with the image in the absence of its associatedtext, and asks the user for the 5 most relevant annotations.The second, called Contextual annotationstandard (GS context ), provides the user with a listof candidates 1 for annotation, with the user free tochoose any of the candidates deemed relevant todescribe the image. The user, however, is not con-1 Union of candidates proposed by all systems participatingin the evaluation, including the baseline system56Proceedings of the Third Linguistic Annotation Workshop, ACL-IJCNLP 2009, pages 56–59,Suntec, Singapore, 6-7 August 2009. c○2009 ACL and AFNLP

Statistical Models for Automatic Video Annotation & Retrieval
Automatic Image Annotation Using Group Sparsity
Automatic Image Annotation Using Group Sparsity
Image Annotation by Hierarchical Mapping of Features
Automatic Image Annotation by Incorporating Feature Hierarchy and ...
Exploring Collaboratively Annotated Data for Automatic Annotation
Automatic Image Modality Based Classification and Annotation to ...
Multi-Label Sparse Coding for Automatic Image Annotation
Automatically Annotating Textual Resources with Human Intentions
Automatic Image Annotation By An Iterative ... - Fudan University
Automatic Image Annotation by Mining the Web - Department of ...
Automatic Annotation of Images from the Practitioner Perspective
Automatic image annotation using a semi-supervised ensemble of ...
Approaches of Using a Word-Image Ontology and an Annotated ...
Automatic Image Annotation using Visual Conent ... - Know-Center
Automatic Image Annotation using Visual Content ... - Know-Center
SIA: Semantic Image Annotation using Ontologies and Image ...
Approaches of Using a Word-Image Ontology and an Annotated ...
Using Heterogeneous Annotation and Visual Information for the ...
Image Annotation Using Metric Learning in Semantic ... - CVIT
Enhancement of Textual Images Classification using ... - CiteSeer
Models for automatic detection of topics in textual data and images
automatic image annotation by input-output structural ... - ijcsmr
Automatic Panoramic Image Stitching using Invariant Features
EXCOM: An Automatic Annotation Engine for ... - ResearchGate
Exploring the use of digital textual, visual and ... - RMIT University
A Framework for Image Annotation Using Semantic Web
Automatic Image Annotation through Multi-Topic Text Categorization
Hierarchical Image Annotation Using Semantic ... - ResearchGate
Semantic Textual Entailment Recognition using UNL