NUI Galway – UL Alliance First Annual ENGINEERING AND - ARAN ...

Improving Categorisation in Social Media using Hyperlinked Object Metadata
Sheila Kinsella
Digital Enterprise Research Institute, National University of Ireland, Galway
sheila.kinsella@deri.org
Abstract
Categorising social media posts is challenging, since
they are often short, informal, and rely on external
hyperlinks for context. We investigate the potential of
external hyperlinks for classifying the topic of posts. We
focus on objects with related structured data available.
We show that including metadata from hyperlinked
objects significantly improves classifier performance.
We use the structure of the data to compare the effects
of different metadata types on categorisation.
1. Introduction
Hyperlinks are often a vital part of online conversation.
Users share videos or photos they have seen and point
to products or movies they are interested in. These
external resources can provide useful new data such as
author information for books, or genre information for
movies. Often the post cannot be fully understood
without knowing these details. Many of the hyperlinks
point to websites that provide metadata for objects, e.g.,
videos (YouTube) or products (Amazon), and publish
this data in a structured format via an API or as Linked
Data. Structured data is useful since it allows relevant
data types to be identified. Fig. 1 gives an example of a
post where useful information can be gleaned from the
metadata of a hyperlinked object. Some data such as the
author is redundant, but the book title and categories
are new. The title and categories can be useful for
classifying the post, e.g., under a ‘Rugby’ topic.
Fig. 1:Enriching a post with metadata from hyperlinks
2. Experimental Setup
We use two datasets, one from a Forum (or message
board) and one from Twitter (a microblogging site).
We identified hyperlinks to sources of structured data,
and retrieved the HTML pages as well as the relevant
metadata. We used a Naïve Bayes classifier to compare
classification based on post content (with and without
URLs), hyperlinked HTML pages, external metadata
from hyperlinked objects, and combinations of these.
We experimented with different methods of combining
91
sources and report the best results. Results shown are
micro-averaged F1 score (± 90% confidence interval).
3. Experimental Results
Table 1 shows results for both datasets. Classification
based on HTML pages alone gives poor results.
Classification based on metadata alone increases F1 for
Forum but decreases F1 for Twitter. Combining
content and HTML pages improves F1, and combining
content and metadata gives even better results.
Data Source Forum Twitter
Content (baseline) 0.811 ± 0.008 0.759 ± 0.015
HTML 0.730 ± 0.007 0.645 ± 0.020
Metadata 0.835 ± 0.009 0.683 ± 0.018
Content+HTML 0.832 ± 0.007 0.784 ± 0.016
Content+Metadata 0.899 ± 0.005 0.820 ± 0.013
Table 1: F1 score for each post representation
Table 2 shows the results of classification based on
individual metadata types in Forum compared to post
content. For posts that link to Wikipedia, the article
descriptions and categories provide a better indicator of
the post topic than the post itself. For posts that link to
YouTube, the video title, description and tags provide
better indicators of the post topic than the original post.
Data Source Wikipedia YouTube
Content (baseline) 0.761 ± 0.014 0.709 ± 0.011
Object Titles 0.685 ± 0.016 0.773 ± 0.015
Object Descriptions 0.798 ± 0.016 0.752 ± 0.010
Object Categories 0.811 ± 0.012 0.514 ± 0.017
Object Tags N/A 0.838 ± 0.019
Table 2:F1 score for metadata types in Forum
4. Conclusion
Our results show that categorisation in social media can
be significantly improved by including metadata from
hyperlinked objects. Different metadata types vary in
their usefulness for post classification, and some types
of object metadata are more useful for classification
than the actual content of the post. We conclude that
hyperlinks to structured data sources, where specific
metadata can be identified and extracted, are a valuable
input for post categorization.
8. References
[1] S. Kinsella, M. Wang, J.G. Breslin and C. Hayes,
‘Improving Categorisation in Social Media using Hyperlinks
to Structured Data Sources’, Proceedings of ESWC 2011.