Where to Find Hidden Intelligence in Unstructured Data

roducing dashboards and generating reports from structured data has been a standard method for extracting insights from accumulated customer information. While this approach has worked well in industries such as online retail with every consumer click, swipe, or purchase stored in a relational database, many sectors, such as healthcare, have troves of unstructured data that sit unused.  

Deciphering signal from this enormous pile of noise presents companies with tremendous potential opportunity for better understanding customer behaviors that lead to better customer experiences.

Specifically, healthcare could see a wave of innovations in preventative care and transformative patient experience based on new intelligence from previously untapped unstructured data. With estimates of upwards of 90% of data in unstructured form, this will be the new gold rush in customer analytics.


Sorting Through the Universe of Unstructured Data

So, what exactly is unstructured data and why is there so much of it?  Virtually every person-to-person communication today produces some form of unstructured data — typically in raw text-based form. Text messages, chat, documents, and emails are the primary culprits. Now with video, audio, and still images the universe of unstructured has expanded dramatically, but it comes in forms even more challenging to interpret in a large scale, automated way.

For example, drones are now used to capture video and still imagines for everything from homeowners insurance claims to measuring customer traffic patterns at shopping centers and amusement parks by photographing parking lot at various times. The challenge lies in evaluating this mountain of data without requiring human intervention — otherwise, it just won’t scale.   

Medical records present a more mundane but potentially transformational example of the power of unstructured data. A typical patient record likely contains a jumble of hand-written notes and documents that may or may not be held within a “certified” Electronic Medical Records (EMR) system. While EMR systems have been adopted by 90% of office-based physicians, consensus estimates are that 80% of that data is in unstructured form. As a result, it remains largely untapped for more proactive and preventative individual care, or for use across patient populations to inform research and public policy. The reason for this is that they are likely stored as scanned image files or pdf documents and not in a relational database that is easy to search, extract and analyze. 

The Opportunity in Healthcare

IU Health is using data analytics to develop a deeper understanding of their patients to improve the patient experience as well as health outcomes. They do this by augmenting their internal data with external data to help spot correlations that could lead to more targeted intervention.  According to Richard Chadderton, senior vice president, engagement and strategy, “It is promising that we can augment patient data with external data to determine how to better engage with people about their health. We are creating the underlying platform to uncover those correlations and are trying to create something more systemic.”  They are looking at correlations like housing density and high disengagement from health to spot areas for early intervention.

By taking this “outside-in” approach providers like IU Health can capture the full context of individual patient behaviors and outcomes. With better understanding, you can imagine a more consumer-like approach to healthcare that would lend itself to use cases such as:

Proactive and preventative Care – patient and doctors with better longitudinal data delivered via shared online dashboards could map trends in vitals, cholesterol, blood sugar, weight, and other factors to anticipate and head off more significant problems. Payers could provide financial incentives for patients to participate, lowering premiums for participation and hitting certain milestones.  Patients with mobile phones could receive alerts and text messages that “nudge” them to continue to monitor their health and adhere to a schedule of preventative care visits that include blood tests and physicals.   

Population-wide data use – By aggregating more granular individual patient data, providers and payers can start to isolate the root causes of broader health problems and develop community-wide preventative programs to educate patient populations and drive behaviors that lead to better overall outcomes. 

The key is to start to get better data on individual patients by unlocking what’s hidden in unstructured form, uncovering trends at the individual patient level, and discovering correlations at the community level that drive innovations in preventative care.  After all, the best outcome is less healthcare and more healthy people.  

Photo credit: Photo by Crew on Unsplash