The following is a contributed article by Emil Eifrem, CEO at graph database company Neo Technology
Article originally posted on IDG Connect
Gate Gourmet, an airline industry catering provider, was struggling to lower a 50% attrition rate among its 1,000 employees at O’Hare Airport in Chicago. Using data easily accessible in internal systems, such as demographics, salaries and transportation options, the company confirmed its suspicion that the attrition rate was directly related to the distance and transportation options from employees’ homes to the airport. This realisation enabled the company to change the hiring process and reduce attrition by more than a quarter.
Gate Gourmet’s experience illustrates the power of Dark Data – non-transactional data that can range from which marketing pieces specific individuals responded to, to what they’ve said about an organisation or brand on social media to customer purchase history, frequency of website visits or geographical spread of customers.
Dark Data isn’t as familiar a term as Big Data, which has become a favourite industry buzzword. Corporate spending on infrastructure to capture and store diverse volumes of rapidly-changing data has risen significantly in recent years as companies have scrambled to collect all of the consumer information they believe will help them stay ahead of the competition.
Typically, most organisations focus their data analysis efforts on transactional data — the information customers supply when they purchase a product or service – because they perceive it to be the most valuable. This typically includes names, addresses, credit card information, etc. However, in the course of collecting transactional data, large amounts of additional customer information also are accumulated as a byproduct.
This non-transactional information is Dark Data. Gartner defines it as “information assets that organisations collect, process and store during regular business activities, but generally fail to use for other purposes.” Dark Data can go a long way toward helping organisations know what is useful and what isn’t, and how to make the insights actionable. Yet, surprisingly, many businesses are not yet leveraging their dark data.
Gate Gourmet exemplifies what can happen when they do. While companies will, and must, continue to actively collect data, it is essential not to neglect the information already available, free of cost. It is clear that there is a need to be more creative by asking new questions from the same old data to throw up exciting and surprising results.
Gate Gourmet did not need to invest a huge amount of money in collecting data to solve the company’s attrition problem. Rather, they needed to look closer at the data they already had available in a way that enabled them to see patterns and connections between employees that were staying with the company and those that were leaving.
Unlocking Dark Data’s power
One key to unlocking Dark Data’s secrets lies in the ability to understand the relationships between seemingly unrelated pieces of information. The way that data is stored plays a critical role in this.
Traditional relational databases, and indeed even many so-called Big Data technologies, simply aren’t designed to show relationships and patterns between data records. You may be able to unearth some connections at a very high level, but the results will be extremely slow and lack real definition. It’s the difference between understanding if two people living in one house are married, siblings or flatmates, and then going a step further to predict how those differences might influence their decisions.
However, discovering the business value of Dark Data is starting to become more straight-forward. NoSQL databases can offer completely new ways of reading traditional data sets. In particular, graph databases such as Neo4j naturally lend themselves to the mapping of relationships between data, making it easy for businesses to see connections between the information they have.
As a result, businesses can ask questions of the database that will bring life to insights in data that have an impact on their bottom line.
The key in monetising Dark Data lies not only in gathering it, but in analysing it to discover hidden patterns, developing hypotheses and then putting the insights to use. Doing this successfully requires a variety of different technologies, each suited to a particular job.
By combining data science and number crunching on large-scale analytic technologies, with the real-time execution of complex algorithms by using a graph database, businesses can bring transformative insights to their operational decisions, and combine the latest technologies with their existing data and systems.
While it may appear obscure and unhelpful, if approached in the correct way, Dark Data can reveal all kinds of patterns and insights that would otherwise have been missed. It is information that can really make a difference.
Want to learn more about graph databases? Click below to get your free copy of O’Reilly’s Graph Databases ebook and discover how to use graph technologies for your application today. Download My Ebook
Keywords: Dark Data • emil eifrem • graph database