What’s all the hype around unstructured data?

Check out the DM Review article by Michael Gonzales “Comprehensive Insight: Structured and Unstructured Analysis” for an introduction into the topic of unstructured data and releasing its potential.

Michael provides some insight into the challenges organizations face in dealing with unstructured data vs. structured and how technology has been evolving to help better leverage such information assets.

With an estimate that “more than 85 percent of all business information exists as unstructured data” it is no wonder that technology vendors are putting more focus on how to extend their products to make this information more accessible and usable.

Although the article gives some interesting insight into the evolution of the technologies it doesn’t provide any insight into how to actually integrate and store this data in the traditional data warehouse. How does one integrate such unstructured data in the form of documents, images, video content, and other multimedia formats? Is such data actually relevant to data warehouses and CPM processes? Perhaps not the actual content but perhaps the metadata associated with the content (e.g. x number of documents types, average occurrence of y in videos of type z, number of emails on subject w, etc).

Vendors that are beginning to address the storage and integration of such unstructured data into existing solutions are primarily the large database vendors. Certainly “Big Blue” (IBM) boasts support for analysis of unstructured data with its DB2 Warehouse 9.5 product offering and Microsoft SQL Server 2008 is touted by Microsoft to “provide a flexible solution for storing and searching unstructured data”.

Although the advances in vendor technologies are providing a means of storing such information in a manner that makes it accessible, is the typical organization yet ready to focus its resources on doing so? When so many have yet to fully realize the benefits of provisioning to the business traditional structured data, e.g. Financial, Operational, Customer, etc, you have to beg the question as to whether this should yet be a high priority?

What is your organization doing? Have you implemented any creative solutions? What are the demands from the business?

Creating an Effective Decision Making Environment

Decisions. Information. Governance. What is the point?

The point is to establish the infrastructure and processes for an organization to learn, adapt, and ultimately make effective and timely decisions. Organizations are constantly faced with a myriad of decisions from the many tactical decisions we make every day to the few strategic, "bet-the-farm" decisions we make less frequently. Decisions are rarely made with perfect information. The real challenge is to create the right environment in order to make the best decisions possible given the circumstances and to ensure that the sum total of the decisions made moves the organization in the direction it seeks to go. Put another way, an effective decision making environment is guided by the goals and culture of the organization and leverages the right technology and processes.

Mark Kozak-Holland wrote a great series of ten articles for DM Review from November 2005 to December 2007 to illustrate this point. The series is titled "Winston Churcill's Decision-Making Environment" and provides a vivid picture of an effective decision making environment. Kozak-Holland takes us back to the early days of World War II, shortly after Churchill has been swept to power and faces a daunting task - how to protect Britain from an imminent invasion by a numerically superior German force with about half of the aircraft needed to defend the homeland.

In the articles, Kozak-Holland does a great job of capturing the key components of the British decision making machine - Bentley Prior (RAF Fighter Command), Bletchley Park (code breakers), Whitehall (fighter supply chain), and Storey's Gate (Churchill's headquarters) - and the tight communication and process interrelationships among the different components. What emerges is a picture of a highly effective decision making environment that is operating under extreme conditions. However, through the use of the leading technology and operational processes of the day, the environment performs brilliantly and allows the British to prosecute the early campaign in a way far beyond the limited resources available.

The articles provide an excellent example of how an effective decision making environment can be a significant competitive asset. While most of our organizations face less dire circumstances than Churchill faced, Kozak-Holland's articles are certainly worth the read.

I would like to hear how you feel the principles raised are evident (or not) in your organization.