Think BR: Don't twist the numbers

Measuring social media reactions to an event is an important way to evaluate a campaign, but a system of benchmarks is crucial to give the numbers meaning, writes Eoghan O'Neill in Brand Republic.

To some, this year’s Diamond Jubilee celebrations were merely an excuse for a four-day weekend. To us at Ipsos MORI and our friends at social media monitoring experts Brandwatch, it was an opportunity for a face-off in a blue-blooded showdown with last year’s Royal Blockbuster, the Royal Wedding. We asked ourselves a simple question: did the Jubilee create more social media buzz than the wedding? Aligning the data on the same time scale (with the key 'zero hour' moment being the flotilla in 2012, and the balcony kiss last year) it emerged that there was more buzz overall about the Jubilee than the wedding over the equivalent four-day period. This is not necessarily surprising since the Jubilee was an extended celebration whereas the wedding only lasted a day. On the other hand, on the peak day of traffic the wedding crept ahead of the Jubilee, with 480,000 mentions compared to 387,000. Jubilee buzz levels hit a peak at 2pm on Sunday during the flotilla and again at the end of the concert the following day. Our verdict on the result was a score draw. A key point here is normalisation of data: this result takes into account the organic growth of social media uptake over time, eg, there are more people on Twitter now than last year so you’d expect volumes to be higher. When comparing social media volumes over a long time frame, it’s important to put a process in place to account for this. This is far from trivial - various factors come into play such as firehose v non-firehose data from Twitter. Of course, this is all light-hearted fluff: whether or not the Jubilee 'beat' the wedding is of trivial interest. Nevertheless, it does demonstrate some business applications - whether analysing a new product launch, the effects of a campaign or an adverse event. From our perspective as a market research company, one important application of social media data is to layer volumes and other quantitative metrics on top of traditional survey research such as a brand health tracker, which can validate the results and help bring survey findings to life. Moving on to something rather sharper, when it was reported that six million LinkedIn passwords were leaked, the wolves were out in force on social media. At the peak of social media activity, mentions of LinkedIn were over seven times the 'normal' figure for the time and day (lunchtime EST on Wednesday 6 June). Manual analysis of a sample of messages gave a richer picture of the conversations going on. Rather surprisingly given the nature of the story, two thirds of social media mentions did not actually express any sort of personal opinion on the matter; the majority were either giving password security advice (40%) or sharing articles from mainstream news sites like the BBC or Mashable (32%). There were, however, some negative comments (19%) - which split roughly equally between direct criticism of LinkedIn and the cynical wit and sarcasm which social media is so adept at breeding. In addition, levels of mentions of the brand dropped to within 20% of their normal levels by Sunday - four days after the story emerged. Compared to, say, BP’s Deepwater Horizon nightmare, this was a relatively short-lived crisis from a social media point of view. Dating site eHarmony also reported a security breach resulting in leaked passwords, prompting one wag to quip: "Wow, big password breaches at eHarmony AND LinkedIn. Someone really, really wants a rich husband." Some commentators argued that LinkedIn’s response was slow and inadequate, but I’m inclined to disagree. There’s not an organisation in the world that would blurt out responses to any old internet rumour without making a few phone calls to check if there might be some substance in it - otherwise they’d be commenting on any old hoax. On the contrary, I’m rather impressed that they responded before the leak was confirmed - taking the opportunity to give practical online security advice, while giving a reassuring message that they were looking into the matter. It is not always a matter of comparing like with like. LinkedIn has a high background level of chat by virtue of its status as a large, mainstream tech brand, so a smaller organisation suffering a similar crisis might expect a much higher spike in mentions relative to their normal level. When setting up a system of evaluation, it important to consider who and what to compare against as the raw numbers will vary widely from country to country, sector to sector, issue to issue and organisation to organisation. It is critically important to ensure that you have a robust methodological framework in place when doing any sort of quantitative analysis of social media data - otherwise numbers can easily be twisted to suit an agenda. The simple fact of the matter is that metrics from social media are only effective for businesses with robust norms to compare against, and when evaluating an event or campaign, careful consideration must be given to choosing an appropriate system of KPIs, and to what extent manual content analysis is required.

Should LinkedIn - or the Royal Household - find themselves in a similar situation in the future, they should be careful to take learnings from previous events, and benchmark accordingly to help solve their business problems.

Eoghan O'Neill is a social listening analyst at Ipsos MORI and wrote this piece for Brand Republic.

More insights about Public Sector