Blog

“Lighting up” Dark Data

Posted on: January 6, 2017 | Posted by: Debjani Deb

Continuing with my latest theme of the use of analytics to enable action (vs. insights alone), I find myself thinking of all the data that is available on digital end-points such as apps, websites, kiosks, ATMs, that fundamentally record user activity as it happens, yet does not enable any real-time action. Most apps today have events flowing into analytics packages such as Google Analytics or Omniture that are used by the enterprises to understand their visitor activity in great detail in order to help design apps, segment and target audiences.
I call these dark data. Why?

Well, these data in most cases are reported in aggregate and not at an individual level. Also, in most cases they are reported with some delay that typically ranges from a few minutes to up to 24 hours. Thus, these sets of analytics remain opaque at the individual level and keep in dark valuable insights, which in turn prevents timely personalized actions. These insights include: what is the user doing right now, in-context, on channel, and what data would be most valuable to take action on so as to provide a personalized in-the-moment experience? This in-the-moment experience can show the user the most relevant content based on their need and context.

So far, individual user activity streams have not been leveraged to present personalized content based on real time activity correlations. While brands have a lot of rich content existing in their various systems, they do not have a way to map these individual activity streams, understand patterns within those streams and present the most relevant content to the user based on those patterns. Stream processing technologies are at a point where it is cost effective to take this type of personalized action while “lighting up” individual activity streams in real time.