Much ado about big data

Submitted on: Wed, 04.16.2014 10:27pm - Gary Owen |

We've been seeing a lot of articles published recently about Big Data. Most of these articles have tended to focus on the limits of big data, especially when the data itself is used to draw conclusions beyond the scope of what the data can truly tell you.

At MITS we've been working with big data since before it was called big data and we find this kind of conversation fascinating (hence the links we've been posting recently). We posted a talk that I gave at a recent conference on the topic of big data and I thought it would be useful to put a few of those ideas on paper to give the talk some more context.

This presentation was to a group of Wholesales Distributors that have come together to work on how technology can help improve their businesses. Since we've been working in the industry for a long time we felt we had something to add to the conversation about big data in this context--one that is different from the ones most other folks use when they talk about big data.

When the big technology players like Microsoft mention big data they are talking about a very different animal than what most of our customers are trying to wrangle. Most businesses are aware that they could make better use of data to run their business, but they often times aren't quite sure how to translate that into a practical project with goals, deadlines, and outcomes. Unfortunately the tools and technologies (like hadoop) that most vendors discuss can only help us analyze these data sets once we've captured them.

Our belief is that making use of any kind of large and complex data set requires solutions that are very industry specific. Without an understanding of the problems facing a company or group of companies it's impossible to know what data to collect or how to interpret it. Luckily for us most of our customers are in the Wholesale Distribution industry so we have been collecting large, complex, well structured data sets for decades. In fact, we focus on building tools and processes to extract meaningful business value from the data distributors collect.

When conversations about big data span different industries they naturally devolve into conversations about tools and technologies. This lowest common denominator form of communication happens because often times the industries being discussed are so radically different in their data needs that underlying technology is the only shared problem. Unfortunately this doesn't do a lot to help spread an understanding of what it means to use big data to draw analytic conclusions for any specific industry.

It is common for non-technical business folks to first think about the tools and technologies when they hear the term big data, and not about the impact it will have on their business. That impact comes from the questions we ask the data, and the conclusions we draw from the answers, which is naturally very industry specific.

I hope you get something out of my "TUG Talk" on Distributors & "Big Data." We look forward to hosting more conversations about how data and analytics can be used to help the Wholesale Distribution industry in the future.