Key trends in Big Data and the opportunity this provides for the Market Research industry

Much has been written about the opportunity presented by Big Data. But how should the market research industry view Big Data, and how should it expand its range of activities to incorporate Big Data into its offer?

1. Characteristics of Big Data

There is not a single definition of Big Data that can be used to explain when a data set is or isn’t actually Big Data. For this reason, the phrase is commonly misused. Frequently, it is used interchangeably with the size of the data set - but by using the term in this way, we are underestimating the importance of some of the other characteristics of Big Data.

So, how can we identify Big Data? There are four main characteristics:

1. Volume. Firstly, let’s dispel the myth: Not all Big Data is big! While it’s true that many Big Data sources require terrabytes of storage space on a daily basis, there are a number of sources that do not require anywhere near this amount. When we analyse Market Research, we typically deal in hundreds or thousands of records, rather than millions or billions. But despite this, Market Research is part of the Big Data universe.

2. Velocity is important. In a rapidly moving world, many Big Data sources are characterised by the speed at which they are moving. A sales database for a global retail brand would be an example of a fast moving database. This could include information about every individual who purchases each and every product provided by that brand globally in real time.
3. Variety is one of the biggest challenges associated with the analysis of Big Data. Because many Big Data sources are unstructured, the range of potential data formats is vast. One of the first phases of being able to interrogate Big Data is being able to identify, classify and understand all of the variables. This is the equivalent of building a data map. The time associated with exploring the data in order to understand it, will frequently be the largest component of analysis time – particularly where the data is in a raw or unstructured format. 

4. Value. It is important to note that not all Big Data has value. And for this reason, one of the first skills any data analyst has to learn when interrogating Big Data is to quickly identify which data are important – and therefore have value.

It is worth noting, that Big Data does not need to have all of the four characteristics given above - but it does need to have at least one of them.

As a result of advances in technology we are now able to deal with three of the four characteristics of Big Data: Volume, Velocity and Variety. This means that the focus in the industry is shifting towards providing analysis to extract Value.

Only by employing analytical techniques effectively, will we be able to extract all the valuable insights that are found within Big Data sets. And this is an art form. Many of the techniques that we use in market research can be successfully employed to analyse Big Data, which is why the market research industry is perfectly placed to embrace the opportunity that it provides.

2. Sources of Big Data

There are three main sources of Big Data.

The first, structured data, will be very familiar to most people in the market research industry, as market research is essentially a structured data industry. Practitioners spend a lot of time designing studies that collect data in a very regimented and precise manner, so that they can derive the insights that our clients require. We think a lot about technical design, question wording and order – and we expect the insights to ‘fall out’ once we have finished a project because we have been clever and thoughtful throughout the research process. Overall, it’s a very structured process.

Examples of other structured data sets that follow similar principles are CRM / EFM databases and 3rd party market databases. All these data sources have a rigid structure as a foundation.

However ... the majority of Big Data is not structured. It is often chaotic, random, disparate and has no obvious patterns. Yet, by volume it is significantly wider, deeper, and richer than market research data. The challenge is extracting the insights from data in this form.

Unstructured data takes two main forms.

Human generated data is content that is provided by consumers across a variety of sources and includes areas such as customer feedback, social media, photos, video and audio. By definition, these data sources all require different analytical techniques than we use in market research.

However, the data source that is growing the quickest is machine generated, unstructured data. It is growing exponentially in volume terms – and is therefore the most interesting to many companies. Examples of this type of data include: weblogs, transactional information, call records, wifi and geolocation.

3. Key trends in Big Data

With the Big Data universe expanding so quickly, looking to the future is very difficult. But there are four key trends that are already happening to a greater or lesser degree that are shaping its development.

Trend 1 - the availability of increasingly sophisticated data aggregation software.

It is now possible to match data from a number of sources.  And by doing this, we are able to understand consumer behaviour in ‘n’ dimensions - rather than the one or two dimensions that we are used to working with. Why is this important? Well, if you can analyse data from multiple sources together, the richness of insight that can be derived grows significantly.

Companies are already at the point where their internal data is regarded as a very important asset. Transactional data, CRM and EFM databases are being used to mine and extract insights that have a dramatic impact on customer experience, company strategy and therefore profitability.

An important development in the market is that it is now possible to mine structured data sources alongside unstructured data sources to merge data sets that were previously incompatible.

Trend 2 – The importance of ‘filtering’ and ‘tagging’

With so much data now being generated, especially unstructured machine data, it is becoming impossible to store everything that is produced.

This means that in the future companies are going to have to think more carefully about which data sources are valuable – and therefore need to be stored – and which can be discarded.  This process is called ‘filtering’, and it will become increasingly used to ensure that only valuable data is available for interrogation.

Once valuable data has been identified, the next challenge is setting up the means by which they can be used in analysis. This is called ‘tagging’. It is now possible to tag data in real time using software, so that analysis can be conducted as soon data is read into the software. This is an important innovation, as it enables users to conduct analysis ‘on the fly’ as soon as data has been generated.

Trend 3 – Monetisation of Big Data sources

Any company that owns a Big Data source which has value is now looking to monetise their assets. Monetisation takes two main forms.

Realising the value of the data collected by selling it in the market. The data collected in the course of companies’ everyday activities often has an intrinsic value. For example, any comparison website has customer journey information that can be mined to understand more about how consumers make decisions. Most companies will have transactional information from their own customers, but this is only part of the picture, so adding additional data collected by other companies helps to build ‘the story’.

Analysis of the data with the aim of improving business performance. This is covered in Trend 4 below.

Trend 4 – An increasing focus of analytics to extract value

Finally, and I think most importantly, the focus in the Big Data space will be very soon on how analytics can be used to unlock the latent value that is held in Big Data sets.

We are just starting to understand the potential of Big Data - and while it has been the focus of much hype over the last few years - this is very much an industry in its infancy. The key to exploiting the opportunity provided by Big Data lies in understanding underlying data structures and then applying appropriate analytics techniques to extract insights.

4. What does this mean for the market research industry?

I firmly believe that Big Data offers the market research industry a significant opportunity to exploit its analytical capabilities. In a market that is contracting in value terms, and where margins are ever tightening, it is one of the few areas that is likely to grow significantly in the short term.

But how should the market research industry adapt to optimise the potential of Big Data? Here are a few ideas on a positioning:

• Market research is already part of the Big Data universe. We need to acknowledge this and behave accordingly

Big Data is an opportunity, and I think the industry realises this. Most importantly, we already have the analytical experience to exploit this area of rich opportunity.

• We must partner with range of data owners / software suppliers. The world is now ready and able to work in ‘n’ dimensions

The key to success, I believe, is partnering with complementary suppliers to optimise the opportunity. We are increasingly starting to work with companies who own Big Data, or who have large market databases. We are also forging relationships with software suppliers in the EFM and Big Data spaces. These types of collaboration will become more commonplace in the future. In all of this, it will be critical for each party involved to put the right steps in place to respect the privacy of individuals and their rights to anonymity.

• We must acknowledge that the data we analyse will not always be in a neat and structured format. We will need to invest time in understanding the data we work with

Data analysis is the key to optimising Big Data. For many of us in market research, we will need to behave in a way that is the polar opposite of what we do today. A new breed of analyst will emerge, who is prepared to invest time and effort getting familiar with unstructured data sources.

In the world of Big Data, nothing is perfect, and analytical creativity is king...