Asia Pacific region - aprc-research

More documents

Recommendations

Info

STATISTICS Something slightly significant If I asked how many clients you had inadvertently misled in the past 12 months, you might be justly indignant. So, I will rephrase the question. How many times have you reported results simply because they were statistically significant How often have you scanned through a set of tables, searching for the little asterisks denoting significant results, reporting findings on that basis How often have you agonised about leaving out results that were ‘almost significant’ How can I use ‘misled’ and ‘statistically significant’ in the same paragraph Everyone knows that testing quant findings in this way is standard practice and is taught in the finest academies. Our own Code of Professional Behaviour, when discussing what a client is entitled to (Section D; point 4), includes ‘… and of the levels of statistical significance of differences between key figures’. You can’t get more authoritative than that. So, what is my concern about statistical significance testing and the way it is often applied in marketing research Does it matter I will point to two issues in particular. One is the logic of significance testing; the other is the poor way it is implemented. We market researchers often work under pressure to complete projects and get the results to a client. Anything that makes it easier for us to unravel the key findings is welcome. So, we look for the little asterisks. We want to tell our clients what is important and what is not. So we report significance. Fine goals; bad technique. As the 1800s morphed into the 1900s, a number of people sought a way to distinguish random ‘noise’ from ‘real’ differences in data. One was William Gosset (the ‘student’ of the famous Student’s t-test who, happily, worked to perfect Guinness stout). Another was Ronald Fisher. They argued about the way a test for a real difference should be applied and the forceful Fisher won out. This left social science inheriting a flawed approach. There are problems with the logic Fisher advocated and most researchers misunderstand it anyway. Indeed, if you ask researchers to interpret what a significance test is telling them, they are usually wrong — and studies amongst groups of academics have shown that the majority of them don’t understand the meaning either. Amazing, I know, but that just shows how a poorly understood idea can take hold and become a part of research tradecraft. Fisher left us with a bizarre idea - that is that there is a specific level of probability at which the whole world changes from ‘no’ to ‘yes’. One outcome is that we researchers are not supposed to take account of the costs of being wrong and the benefits of being right when deciding the worth of a result. If you were researching the efficacy of a cancer cure, would you think the balance of the costs of being wrong with benefits of being right are the same as when you test the preferred flavour for a new ice cream Fisher’s logic says both situations are to be treated in the same way. An implication is that we researchers can’t take account of clients’ differential willingness to accept the consequences of uncertainty in research findings. All clients, all risks and all benefits are the same. In addition, whatever the logic, we do not give clients the level of skilled service for which they think they are paying when we: • Carry out significance tests after the data are in, missing the point that if we did Power Analysis before the research was conducted we would have known that there was no way the sample could deliver a significant result. Might a client make a case for deceptive and misleading conduct or negligence • Ignore the fact that whether or not a result is significant is a function of the sample size. This means that we might report ‘significant’ findings when they are in fact trivially small. This makes research look disconnected from the realities of the commercial environment. • Don’t report the confidence intervals for results, hence not giving the client (or ourselves) the chance to see just how large a difference really is in the context for which the findings are to be used. Rob Hall, director, Environmetrics There will be a NewMR virtual event that explores text analytics. Titled ‘Listening is the New Asking!’ it takes place online on 8 March. To find out more visit http://newmr.org/page/ listening-2011. Research News will look at text analysis in more detail in the April edition. If you’d like to contribute, email editor@amsrs.com.au 20 Research News March 2011
THE WORD’S OUT Language, social media and thought Social media is all about language and the written word. People are writing more than ever before with status updates, tweets and blogs. There is now a constant stream of text about what we do and think. Of course this is a huge opportunity for market research. The question becomes how to process the never-ending streams of text available on social media. Computers are good at calculations, but text analysis has always been the problem child. Language has a complexity that simple numbers can’t capture. Two schools of thought seem to be crystallising as to how we analyse social media. The first emphasises what is generally called ‘natural language processing’. Using this approach comprehension comes via computerised parsing and analysis of the text based on our understanding of language. Grammar is the key. The second approach is statistical. It emphasises analytical techniques such as Latent Semantic Analysis (LSA), which uses the statistical properties of text to glean meaning from it. Text analytics is the new frontier that market research must conquer. The sheer volume of text can’t be ignored, but it can’t be coded and tabulated the same way open ends are today. There is simply too much. In some senses the contest of grammar versus statistics has already been decided. When you look at text from social media it isn’t grammatically correct, because we don’t speak and communicate in formal blocks of text using perfect grammar. Hence the grammar route is fundamentally at a disadvantage. The argot of web may well defy parsing. It’s significant that search engines such as Google and Yahoo use techniques very close to LSA to process documents and decide how they are related. The core metric of LSA is word counts and co-occurrence of words in documents. Given that search engines are the tools that have made the web usable it is worth paying attention to this. The problem is sentiment. One of the critical tasks for analysis of social media is deciding if a tweet or status update has a positive or negative tone. Is the buzz about a product good or bad If you can automatically tell the sentiment of comments relating to products or services you have a powerful research tool. Sentiment analysis is currently a hard thing to do accurately. Anything over 75 per cent accuracy in sentiment analysis is currently regarded as pretty good, but this still leaves a large margin for error. At the moment the only accurate way to assess sentiment is for a human to read the text, but there is simply too much text being produced by social media for this to be practical. Sentiment analysis of social media is rapidly becoming the ultimate goal for text analysis and market research. We need more sentiment to master social media. Andrew Jeavons is a member of Nebu, which is working to bring innovation to survey software. You can contact him at andrewjeavons@nebu.com This article was first published in Research World, the magazine for marketing intelligence and decision making published by ESOMAR. For more details go to http://rwconnect.esomar.org Ask around. You’ll find West Coast Field Services is your best WA field provider - and has been for over 20 yrs. These days, just about anyone with a database claims to offer market research. But surely your clients deserve quality data that they can rely on For professional Western Australian field resources , you need West Coast Field Services (WCFS). Contact us for CATI, F2F, online, mystery shops, group recruiting and group rooms. We have been WA’s premier research field resource for over 20 years, and are the only WA field company that is purely dedicated to market research, - and to our clients of course! The WCFS team members are long standing professionals in market research. All interviewers are trained to ISO 20252 standards. (Exceed them actually). Our 20,000 strong database recruited from random dial surveys – no customer or telemarketing sales lists. Foundation AMSRO members in WA and continue to uphold AMSRO awards and standards. Our 20,000 strong groups database is morphing into WestSense, the only WA focussed pure market research online panel. Where integrity still matters. Ask around. Contact Sandra Simpson for your West Australian field needs. 08 9316 3366. Sandra@wcfs.com.au. www.wcfs.com.au
Page 1 and 2: ResearchNews Australian Market & So
Page 3 and 4: Research News March 2011 ASIA PACIF
Page 5 and 6: A passion for excellence. A keen co
Page 7 and 8: NEWS The organisers of APAC 2011 no
Page 9 and 10: No need to pack your bags, we can c
Page 11 and 12: fact: Size Matters The larger the t
Page 13 and 14: WRITE TO REPLY Talk (i.e. gossip) a
Page 15 and 16: Dominic Carter Bob Chua James Fergu
Page 17 and 18: FEATURE Current market research awa
Page 19: FEATURE duck because I say, “Now
Page 23 and 24: NARRATIVES implementing those came
Page 25 and 26: CAREER MOVES Tim Jennings To: Group
Page 27 and 28: CAREER MOVES CONTINUED FROM PAGE 25
Page 29 and 30: HR, LEADERSHIP & MANAGEMENT SOCIETY
Page 31 and 32: Over 3 million pre-screened online

Asia Pacific region - aprc-research

Create successful ePaper yourself

Delete template?

Save as template?