Little Insights About ‘Big Data’


A little over a year ago I became the CEO of a company in the “big data” space that is developing predictive analytics software. My CTO is the “data guy,” and I’m the “business guy.” While my background is technical, I had very little experience in data or analytics and only knew about big data from repeatedly seeing it in the press.

I thought I’d share some of the insights over the past year that helped me wrap my head around all the big data hoopla. I find it impossible to learn something unless I have a mental framework on which to hang new concepts. Without a good teacher to provide one, half the battle for me is creating that framework. I’m hoping that by sharing what I’ve come up with, others can get up to speed a bit more quickly.

These insights are simplified distillations that allowed me to put the things I was hearing into a useful context. This allowed me to talk to others about the big data concepts in my own words and sound like I knew a bit about what I was saying.

The first question I had was why the boom of big data is happening now. I concluded it is because of a number of technological changes arising around the same time and colliding in a perfect storm:

—Due to the Web and growing “Internet of things,” the number of companies with “big enough data” has exploded into a market big enough to be worth addressing by many solution providers.

—Cloud technologies are available and affordable, allowing development of solutions that require computing resources beyond the scale most companies can afford to manage themselves.

—Leaders in the big data space, such as Google, Amazon, Facebook, and others, have developed solutions for handling big data and made them available as open source software.

My first insight was to realize that today the label big data is, for the most part, a vague marketing term used to describe any product that interacts with data in any way at all. It is important to belong to the big data club, so if a company can plausibly claim that its product is a member, it will do so. One company’s definition of big data may have no relationship to another company’s definition.

Many big data products are “just” traditional business intelligence, visualization, statistics, or database applications modified to scale to work with much larger data sets without really changing their functionality. I put “just” in quotes because making these changes is often quite difficult. However, the conceptual capabilities are not new, despite the claims their marketing departments might make.

I found that mentally replacing … Next Page »

Single PageCurrently on Page: 1 2

Art Mellor is a software engineer at Skelmir. Previously, he was CEO of Zero Locus, a Milwaukee startup now operating as Functor Reality that creates predictive analytics software for large data sets using probabilistic graphical models. Follow @

Trending on Xconomy

By posting a comment, you agree to our terms and conditions.

3 responses to “Little Insights About ‘Big Data’”

  1. Gabe D says:

    Thanks for an explanation we mere mortals can understand. Your 3Cs really helped clarify the picture

  2. DataH says:

    Great insight Art. It is worth mentioning the HPCC Systems open source offering which provides a single platform that is easy to install, manage and code too. Their built-in analytics libraries for Machine Learning and integration tools with Pentaho for great BI capabilities make it easy for users who do not hold a PhD degree or carry a title like “Data Scientist” to easily analyze Big Data. I believe HPCC is better than Hadoop and commercial offerings, it has a real-time data analytics and delivery engine (Roxie) and runs on the Amazon cloud like a charm through the Instant Cloud portal. For more info visit:

  3. McGee Young says:

    Nice insight Art!