Tag Archives: Knowledge

Big Data or Smart Data?

The hot topic at any marketing, insight and data analytics conferences at the present time is Big Data. In our increasingly technological driven world we are generating enormous volumes of data, some estimates suggest the world is adding several quintillion (1018) bytes of data each day. And as marketeers and analysts we see this as a goldmine of information and we are instinctively driven to find ways of making use of it.

In terms of marketing activity, “Big Data” relates to the ever growing data sets created by our customers and prospects, not just through the traditional marketing database or CRM, but also from retail and eCommerce activity, from social media sites such as Facebook, Twitter, LinkedIn, Flickr and Foursquare. We can also access data from sources such as Google Analytics where we can see how visitors find our websites and what they then do once there, and media can include the full range of digital formats such as pictures, video,audio, CCTV.

For example a financial lender may wish to  look at an applicants Facebook site as part of the credit checking processes to see if they have pictures of holidays every other month and parties every night – would they be a safe bet to lend to?  We have all heard stories of employees checking out social media sites of potential employees as to their suitability.


Would you give a pay-day loan to someone boasting this shoe wardrobe on Facebook?

The immediate consequence of wanting big data is that you need big servers and big software to process it and at this point the IT Team takes over, rub their hands with glee and an obsession develops on gathering all the data that can be found. Little consideration is given as to why the data is being collected, other than “it will help our marketing to be more effective” and planning what is going to be done with it once its gathered is often non existent. And with this data hunter gatherer approach, it seems that the sound statistical and analytical approach to extracting knowledge and insight disappears and in consequence it becomes easy to produce results from looking at the data that at best can be skewed an,d even worse, can be totally unsound. And there is also an implication that big data with its big servers and big software will cost big bucks so making it an exclusive tool for the large multinational Company, leaving us small and medium enterprises behind.

I would argue however that what is really needed to make best use of the data available is not Big Data but Smart Data. Concentrate on looking carefully at what data is available, extracting only data that is potential directly relevant and then using sound statistical sampling techniques to analyse and unlock the insight and knowledge within the data.

In a sense the term “Big Data” is a strange term as it is used to imply that size is in some way a measure of quality and value and I believe in a few years time we will find it odd to talk about big data. Size in itself doesn’t matter – what matters is having relevant data that helps us solve a problem or address the questions we have and so I prefer the term “Smart Data”. Smart data is available to all organisations small and large. It can use existing tools and techniques to evaluate and gain insight.  And when we want to bring in the additional data sets that the likes of social media and Google Analytics offer us then the way to do this is through creating and integrating small data sets using the techniques and systems we already have in place and not through building big data monoliths and creating massive centralized data warehouses.


Data, Intelligence & Knowledge Part 1 – Data Management.

At the heart of any direct or digital marketing communication is data. And all companies or organisations, large or small, who wish to communicate directly with customers or prospects collect and store data. But my experience of dealing with hundreds of businesses over the past 25 years is this core activity is usually undertaken with little planning or forethought, is executed in a haphazard manner and very rarely is the potential that the data offers the business unlocked to the full benefit of the business.

The first fundamental error too frequently made is deciding where responsibility for data management resides within an organisation. Too often it is seen to be an IT function because the initial challenge facing an organisation is how to hold the data. One of two approaches is taken:-

  • The Organisation is already collecting customer and prospect data in its accounting system, order system or other back office facility and so there is no immediate need to consider the marketing function. The organisation is “sure” it can extract the data in a usable form when it’s needed by marketing.
  • The Organisation decides to buy in or build a data repository for marketing activity. Whether it’s an excel spreadsheet, a marketing database, a data warehouse, a CRM system or a Big Data model, IT will thrive on creating the container to fit the data and spend significant budget building something that too often will constrain what data can be stored and how it is held. It’s an approach similar to the food manufacturer designing the packaging and then deciding how to cram the product into its container.

The second fundamental error is the belief that once the data repository has been built then the hard work has been done. In practice the effort and resource required to manage the data far outweighs the effort to build the repository in the first place.  I see far too many databases or CRM systems that fail to reach anywhere near their potential or on occasions don’t get past day one because no resource has been allocated to manage the data collection and on-going maintenance. 

The starting point for any organisation that wishes to undertake direct or digital marketing is to formulate a data strategy that clearly identifies what data is available to the organisation, what data needs to be held and how this data will be used going forwards. From this it becomes easier to define how the data will be held and gathered, and the resource and effort needed to undertake this function. And as with any business strategy, it needs to be reviewed and enhanced at regular intervals going forwards.

So what are the key components of a data strategy?

  • What customer or prospect data is available? You obviously need contact details, a fundamental for communications but what other data is known or could be available. It could be transactional/purchase data,  a history of communications from and to the customer or prospect, lifestyle data (B2C) or Company data such as turnover, employee numbers (B2B).
  • What data should we store and retain? The temptation is to keep all data but this can create a significant resource requirement in terms of storage and maintenance, particularly when social media data is involved. The Data Protection Act also requires us to only hold personal data that is relevant. Smart data”is better than big data.
  • How much data are we likely to gather? This is key to determining the data storage vehicle and the resource needed to maintain the data going forwards.
  • How will we store the data?  The choice here is endless and much depends on the answers to the above questions. And there are also decisions to be made as to whether off the shelf software is available that will match the data requirements (low up front cost, possible compromise as to what data is held and how), whether to produce a tailored in-house system (high up front cost, but system matches the data requirements) or whether to put the data management out with a third-party specialist (low up front cost but possible higher running costs).
  • How will we collect the data, input into the system and then maintain it? Included in this is the question of data quality. For example data trawled from the web or social media that is input by the user is often of poor quality, badly typed, abbreviated and difficult to utilise without significant data cleansing.
  • How do we keep the data up to date? People die, they move, they get married, they ask not to be contacted. Again the Data Protection Act requires us to keep personal data current.
  • What are we going to do with the data and will it serve this purpose?
  • How long will it take to build enough data to make it viable for the applications required of the data? Good data is like good wine, it comes into its own over time and with the right care will mature. Often databases will take several years to really start performing at their full potential.

But this structured approach to data planning is not what I see with the majority of UK Companies today. Running a Company that provides a full range of data strategy, management and analysis services, DDL Group spends more time with Clients sorting out bad data to make it accessible and usable for marketing campaigns than it does helping Companies with good data understand and unlock the power of the knowledge within the data to bring better ROI from its campaigns.

Only when your data model is correct will you be able to start unlocking the full potential of the intelligence held with the data which in turn will generate knowledge as to who your customers and potential customers are and more importantly how and why they buy from you. The ultimate aim of any marketeer is to have actionable intelligence about their customers that will help improve the customer relationship, increase retention and ultimately improve the return on marketing investment.