Big Data – big problem or big deal?

McKinsey’s coining of the term ‘Big Data’ has given a name to something with which marketers have been grappling for some time.  McKinsey’s recent study reported that, back in 2007, the amount of digital data created in a year exceeded the world’s data storage capacity for the first time; there was no way to actually store all of the digital data being created then and has there has been no going back since.

The mantra or DM has always been that it is measurable – that we have in effect solved the conundrum of unlocking value from data. However, as datasets proliferate and increase in size, we all become potentially less comfortable – does the sheer volume of data mean that we don’t know what we don’t know?  Does it mean that the datapoints that we have always treated as gold dust may now have been superseded?  Most importantly, should we be nervous of the fact that we will probably never access all of the data that relates to our customers and prospects?

McKinsey’s view of the subject has in mind issues far greater than the commercial imperatives of marketers of course.  They point out this proliferation of data can have an impact on areas as diverse as social services, public health and law and order. They also acknowledge that there is an underlying caution or even suspicion felt by consumers that this is all becoming rather too Big Brother;  that someone somewhere may be taking liberties and making profits.  This is the very concern that has impacted many new areas of marketing – from mobile Location Based Services (LBS) to online Behavioural Targeting.   It is the very same concern that exercises Governments and legislators (the most pressing example of which is the EU’s revision of its Privacy & Electronic Communications legislation.)

Great digital pioneers – not least Sir Tim Berners Lee – see the potential to turn this mass of data into a positive power for the consumer.  There are already examples of this such as www.data.gov.uk which provides consumers with a range of local information based on data sources that would once have been strored and accessed behind closed doors.    McKinsey sees this as the tip of the iceberg and that further developments will be much more disruptive and may cause fundamental power shifts in some industries – they cite medical clinical information providers that aggregate and analyse data; PwC recently identified the very same trend in ‘infomatics’ (which it defines as ‘identifying valuable data and translating it into information to customise patient care and achieve the best possible outcomes’.)

In the commercial sector, McKinsey identifies the intelligent use of big data as being one of the tenets of Tesco’s expansion from its supermarket foundations into financial services and any number of additional markets. The UK’s flagship retailer has long since been the poster child for intelligent data mining based on loyalty cards but now it seems many other sectors are seeing the principle applied elsewhere.

Mckinsey recognises that many of the underlying disciplines necessary for realising value from Big Data are familiar to organisations – it cites segmentation as an established element of both marketing and risk management but sees the same discipline as being new to the public sector.  Those pioneers of the use of census data may well disagree – much of the original analysis that led to the creation of geodemographics was based on exactly that principle; what is different is the mass of data and the frequency with which it is updated and expanded.  We no longer live in ten year cycles. For marketers, real time targeting is now a very real expectation.

Today, digital marketers are already grappling with vast volumes of data that need to be filtered and analysed to meet the current everyday objectives of targeting, measurement and attribution.  Nowhere is this more marked than in the area of attribution where practitioners are bringing together datasets from channels including  display,  search, social and email – a dynamic mash up of data, some PII and some anonymous – from which the gold dust of ROI is sought. How often is our enthusiasm tempered when the reality of gleaning actual value from vast data volumes takes hold?  It is this dilemma that Big Data promises to exacerbate – and the solution that we as marketers must deliver.

There is no single channel in which this growth is better illustrated than social media.  Making sense of the vast amount of ‘noise’ and then assigning a value to efforts to create it remains one of the key challenges for most markets today.  And it is ‘most markets’.  Any view of this as a ‘channel for the young’ has been brushed aside – McKinsey’s survey showed a 2009 increase in usage of c 21% among those aged between 35 and 54 and a 52% among those aged between 55 and 64.

And who will address these issues?  McKinsey points to ‘the shortage of analytical and managerial talent’ as a ‘significant challenge’ that needs to be addressed ‘in the near term.’  It is estimated that skills demand will exceed supply by 50 or 60% over the next 6 years in the US (and there is little to suggest any marked difference in Europe.)  In this context, commercial organisations and public bodies face a real imperative to properly characterise the work of analysts and consultants – this is key work that is challenging, intellectually stimulating and must be financially rewarding as a career path.

The rise of big data will exacerbate today’s very real marketer challenges in creating CRM and targeted marketing – lack of a Single Customer View, targeting restrictions, frequency demands, channel conflicts etc.  Of course all of this organisational change, resource and budget commitment will have a bottom line.  For any of us who have ever grappled with the difficulty in placing a P&L value on data assets, Mckinsey’s views seem prescient – ‘organisational leaders often lack the understanding of the value in big data’ and ‘many organisations do not have the talent in place to derive insights’ or the ‘structures, workflows and incentives in place to make better decisions and take more informed actions.’  Of course achieving this is easier said than done – no one suggested that it is simple.  It’s unlikely to become any more so.

McKinsey cites ‘marketing levers’ as influencing 10 to 30% of operating margin.  Marketing therefore has to have a place at the top table.  In this context, big data is a big deal.

Nick Fuller, Director of Planning and Development, e-Dialog, looks at the implications of ‘big data’ for marketers