BI SEARCH AND TEXT ANALYTICS - The Data Warehousing Institute

BI SEARCH AND TEXT ANALYTICSUnstructured data mustbe transformed intostructured data beforeparticipating fully in BI.We need now to extend that assumption to also encompass unstructured data and semi-structureddata. They also require extraction to locate relevant entities and their facts—followed bytransformation into appropriate data structures—before they can be loaded into a data warehouseand be useful for the traditional accoutrements of BI, like standard reports, multidimensionalanalyses, and statistical analyses. The curious irony is that this data is unstructured or semistructuredin its source form, yet must be transformed into structured data—via some kind of textanalytics—before participating fully in BI.A definition of text analytics. Text analytics is where a software tool parses text and extracts facts(addresses, parts, complaints) about key entities (customers, products, accounts). Recognizingentities and facts about them involves natural language processing (NLP), which “is a subfield ofartificial intelligence” that “converts samples of human language into more formal representationsthat are easier for computer programs to manipulate.” 1 The facts and entities extracted via textanalytics may be stored in a file, database, or search tool’s index. Hence, text analytics (sometimescalled entity extraction) imposes structure on information found in unstructured data sources andsometimes semi-structured ones.Text analytics always involves the transformation of unstructured data into some kind of datastructure. It may also take the next step, which is to provide an analysis of the resulting structure.Or another tool may effect the analysis, based on the data structure created by text analytics.The State of BI Search and Text AnalyticsIncorporating unstructured data and search technologies into BI sounds good. But do userorganizations think it’s worthwhile? Is anyone really doing it?TDWI asked survey respondents to rate the business value of BI search and text analytics as theyunderstand them today. With eager zeal, respondents rated the business value as high or veryhigh for both BI search (62%) and text analytics (51%). (See Figures 2 and 4, respectively.) Thisebullient bubble bursts when we consider their response to the next question: “What’s the status ofyour organization’s use of search in BI” and text analytics? (See Figures 3 and 5, respectively.) Veryfew have deployed either BI search (9%) or text analytics (6%). And relatively few have committedto a design or implementation phase for BI search (15%) or text analytics (7%). It appears thatthese practices are still very new to BI, so few user organizations have deployed or started buildingsystems for them.TDWI suspects that adoption of both BI search and text analytics—though rarely deployed today—will increase over five years, until they are as commonplace as Web GUIs and dashboards are today(although these were rare five years ago). Of the two, BI search is currently in the lead and willmost likely enter the mainstream first.USER STORYBI users are aware of BIsearch and look forwardto using it.“We’re like a lot of companies, in that we’re looking into BI search, but we haven’t done anythingyet,” said the general manager of a performance management group at a large financial services firm.“For years, we’ve been using two major BI platforms; one for general enterprise use and the other forfinancial analysis. Both have BI search embedded, so our plan is to start there with a pilot programand see what happens. I bet that once we turn this on, users will love it so much that we’ll have togive them more.”6 TDWI RESEARCH1 Source: Wikipedia at http://en.wikipedia.org/wiki/Natural_language_processing.