The Price You Pay When Your Data is Questioned

I read this article in yesterday’s Wall Street Journal that I found interesting and relevant to DIG. There is always a debate on the value of having “one version of the truth” and the necessity of accuracy in corporate data. To date, that hasn’t been the case with certain types of performance measurement, especially website visits. Well, comScore is paying the price through shareholder value and their stock price. The issue stems from the accuracy of “clickstream” data that comScore, like their competitor Nielson, collect and track the popularity of websites on the web. Google announced that advertising clicks grew by 20%, while comScore reported only a 1.8% growth. Well, who is right?

This data is critical for marketers when deciding where to spend their ad dollars. You should read the full article to gain a full appreciation of the entire story, but here are a few snippets that are relevant to the importance of having “one version of the truth”.

Sarah Fay, chief executive of both Carat and Isobar US, ad companies owned by Aegis group said “We have not expected the numbers to be 100%”. It’s good to see that no expectations were being set out of the gates. Not sure this would fly when discussing something like revenue for an organization.

The article goes on to point out that comScore and Nielson data doesn’t always match up. “To complicate matters, disparities between comScore and Nielson data are common, as the two companies use different methodologies to measure their audience panels.” This isn’t something we don’t here inside the four walls of a corporation for something like a measures calculation rule.

Brad Bortner, an analyst with Forrester Research points out “There is no truth on the Internet, but you have two companies vying to say they are the truth of the Internet, and they disagree.”

And finally, my favorite quote in the article came from Sean Muzzy, senior partner and media director at digital ad agency http://www.ogilvy.com/neo/. “We are not going to look at comScore to determine the effectiveness of Google. We are going to look at our own campaign-performance measures”. This would be the equivalent of “if you don’t like the results, try a different measure.”

I have always wavered on the need for accurate data for certain types of measurement, especially something like clickstream analysis. I guess that wavering has now fallen to the side of the camp with the other types of data that require precision and accuracy.

Immelman crowned Masters champion!

Congratulations to South African Trevor Immelman who secured a maiden major title win today with a three-shot victory at the 72nd Masters Golf tournament at Augusta.

If you are a golf fan it was for sure a quality weekend in front of the TV.

If you were immersed in the golf, waiting for Tiger to yet again make a run for the championship, did you ever find yourself wondering where on earth do the commentators get all those performance statistics from that they continuously feed you during their commentary?

Well check out the article on “How the PGA Tour Manages Its Data” to see how “IT Chief Steve Evans relies on legions of golf-crazed volunteers, high-tech lasers and the input of golf pros to help him identify, manage and display the Tour's most critical data.”

Thomas Wailgum’s article provides interesting insight into the efforts made to capture data real-time from the field, then to translate it into interesting statistics that both the general public can hear about to enhance their golf watching experience, as well as players on the course to not just help them track their progress but also help them evaluate their risk exposure when contemplating their next shot.

The system used is ShotLink, a revolutionary system that “tracks every shot at every event—where a player's golf ball starts and lands, and all the ground covered in between.” ShotLink requires the use of over 1000 volunteers out on the course to help “capture” the required data on over 32,000 shots which is feed into the system.

Impressively in the article Mr. Evans states that through some minor modification to ShotLink they have been able to ensure a very high level of data quality in the statistics that they produce. He states that “Our goal is to have any data corrections made inside of one minute, and we consistently meet that metric.”

How accurate is the data that you use? What data quality manaagement process do you have in place? Do you tend to resolve data quality issues upstream at its source, or do you cleanse within the applications your report from, or "tweak" the actual reports?

Opening Day (Part 2)

To help me get ready for the upcoming baseball season, I recently purchased the 2008 Baseball Prospectus, an annual almanac of analysis and predictions from the folks who brought us 21^st century metrics like VORP (Value Over Replacement-level Player) and BABIP (Batting Average on Balls In Play).

I got to thinking about the evolving perception of analytics in a baseball context. While mentioning those metrics at the water cooler of 2008 might get you some confused looks, you're probably not in danger of being stuffed in a locker, because analytics have begun to earn respect from a wider baseball audience. How, exactly, did this happen? Off the top of my head, I came up with four possible drivers of this change (not mutually exclusive):

1. Boil the frog slowly – If On-Base Percentage really correlates better with runs scored than batting average, well, who am I to argue? And if that's true, then maybe I ought to listen to some of your other ideas...


2. Myth-busting – Some assertions have been controversial (e.g. “There’s no such thing as a clutch player”), but maybe they’re plausible and interesting enough to get attention


3. Case studies – Michael Lewis’ best-seller Moneyball and the 2004 and 2007 Boston Red Sox (World Series winners) have shined a public light on organizations that succeeded with analytics-friendly leadership


4. Audience evolution – People in general have better quantitative reasoning skills than they did, say, 20 years ago, and so are more open to evidence-based insights

Here’s my question, and it’s not about baseball: In your organization and mine, a major barrier to extracting value from analytics is a rejection of the methods and implications from (for lack of a better term) the “old school” crowd. What’s the best way to make the case for analytics in your organization? Take on a single cherished nugget of conventional wisdom and prove it wrong? Or is that too risky? Is it better to plug along cautiously, incrementally adding some objectivity and trickling new metrics into the soup until the organization is ready? Or is it the Moneyball approach – find one manager willing to try the Kool-Aid and make something happen?

Opening Day (Part 1): Roger Clemens is Innocent

I’m excited to join such a distinguished group of bloggers, and proud to reveal a new analytic insight in my first post.

It seems to be the consensus among baseball fans that Roger Clemens used performance enhancing drugs in the latter stages of his Hall of Fame career. Accusers support their claims by referring to his apparent improvement when he left the Boston Red Sox for the Toronto Blue Jays before the 1997 season, right before his age-34 season. But advanced analytics tell a different story altogether.

Using some new functionality in the latest version of SAS, I created two new metrics. One is something I call Adjusted Prevented Runs (In League). Basically, it controls for a handful of factors not addressed in the pitching metrics most favored by sabermetricians today and in one number tells you how effective a starting pitcher is, relative to the rest of the league, in keeping the opposing team from scoring. I call the second metric Factored Outs Over League, because in terms of pitching performance, while keeping opposition runs down is the ultimate goal, the way to achieve it is by getting outs. It’s like a pitcher’s version of On Base Percentage.

Because of the relationship between preventing runs and getting outs, the best way to get an overall view is to multiply the two metrics. And when you look at the data, Roger Clemens didn’t actually register a significant change in his performance from his Boston years to the Toronto years.

Thus, with the help of APR(IL)*FOOL, the power of analytics is demonstrated once again: this time, to clear the name of an innocent American.

On the road in Miami

I am on the road this week down in Miami attending a Palladium conference on "Developing a Dynamic Strategy". I felt compelled to do a quick post on one of the speakers this morning since there were some interesting points that he made during his presentation. The image to the left is the Biltmore Hotel where the conference is being held. It is a very historic hotel with a great golf track where Bobby Jones would frequent back when golf clubs were actually made of materials like wood!


Ralph LaRossa from PSE&G gave an entertaining overview of how his organization links their strategy to the operations of the business. As a utility company, PSE&G has a focus and strategy built around areas like safety, customer focus and environment. They seem like pretty basic concepts, but what was interesting was how they actually tied these strategies to operational metrics. For example in the area of safety, they track number of incidents and duration of leave for an employee associated with accidents. PSE&G then takes it one step further and measures themselves against industry benchmarks in the utility industry. They use their measurement system across the entire organization to drive visibility and accountability. PSE&G shared positive quantifiable results across their measurement system and have become leaders against their peers.

From a compensation perspective, Ralph provided insight into how employee contribution and compensation ties back to the performance of the organization. They have a well defined compensation system that makes ultimate company performance a function of how individuals will be compensated. This allows everyone from the field to understand how their role and how measures associated with safety and customer satisfaction impact the strategy and ultimately the performance results.

Like I said, Ralph was very entertaining. He had some funny anecdotal stories about being a company based in New Jersey and made many references to The Sopranos. During the Q&A at the end, I wanted to get his thoughts on the final episode, but I thought better of it.

Real-time Data Usage

This past weekend was a major weekend for me, one that would determine my core happiness for the rest of the year!

You see I’m Rugby mad. I’ve played it, I’ve coached it, and now I watch it, incessantly.

This past weekend was the concluding weekend of the European 6 Nations Rugby Championship, an annual international tournament between the home nations of Europe (England, Ireland, Scotland, Wales, France and Italy). A tournament that started in 1871 and every year since has been an excuse for all to bring out their nationalistic pride and cheer for their ancestral team! For me it’s Wales, the land of my forefathers, the land of daffodils and of course sheep. Wales had the opportunity to win the tournament in style, beat France at home and raise the championship trophy undefeated, Grand Slam winners. And did they do it? They sure did!

Rugby, just like most American sports is now a professional sport and with it many changes have come. Dragged out of the traditions of amateur pastimes where the local butcher was your star player, teams today are forced to continually explore all possible avenues in an attempt to better themselves and obtain competitive advantage over their rivals. No longer is it good enough to just employ the best players and coaching staff, teams are looking elsewhere.

One interesting field of innovation that we were given a brief insight into during one of the games was the usage of statistical information real-time by the Welsh coaches that allowed them to then make real-time adjustments to how their team and players were approaching the game. Using the interesting technology Sportstec the Welsh team was actively making adjustments that helped provide a competitive advantage over their opposing team.

Around the field “spotters” were employed to feed information into a database application on specific events happening. Number of times a player passed to his left vs. his right, how many carries an individual had with the ball, number of times the ball was kicked from a certain place vs. passed, number of missed tackles by each player, success rate of a particular move, etc... By providing such detailed information on actions performed by their team as well as the opposition, the coaches were then able to react and make tactical real-time changes, for example adjustments to the team’s strategy on the field, instruction to specifically focus and improve in certain aspects of the game, as well as instruction to target identified weaknesses in the opposing team.

Did having this level of information access have a direct result on the outcome of the game? Who knows, but one thing for sure Wales beat Italy in this game 47-8, when on average the other teams who beat Italy did so by only 6 points! The other thing, did I mention, Wales won the Championship, undefeated!

So where next? If teams are able to get hold of and make use of such real-time data I just wonder how much further they could go.

What if players began to wear RFIDs in their shirts so we know where they are at anyone time. The ability to understand in real-time how much time they have spent in one location, how much time it took them to reposition, the average distance they make while running with the ball, efficiency of path travelled by each player when covering a kick-off. What about collecting information from body skins that can sense applied pressure? Could we measure the level of impact endured in a tackle thus giving the ability to predict level of fatigue vs. amount where performance begins to degrade, an opportune time for a tactical substitution perhaps?

For more detailed commentary on how the Welsh team and others are finding innovative ways to capture and use information check out the videos @ http://www.sportstec.com/videos.htm

Providing visibility into municipal performance

I was recently forwarded a link to an interesting New York City online reporting tool. The purpose of the tool is to provide visibility into the city's service performance. As a diehard Boston Red Sox fan, it is a tough pill for me to swallow with the next statement I am about to make. This is a pretty cool system for New York city and a government agency to develop.

The City Performance Reporting (CPR) tool provides performance results and outcomes organized by 8 citywide themes, such as Education and Public Safety. The purpose of CPR provides transparency and accountability for city services to the "users" of these services...New York city residents.

Some of the features that CPR provides includes aggregation of metrics, year over year performance, traffic lighting for variances, metric definitions and graphical representation of data. One additional feature that I thought was interesting was the general trends up or down for the aggregate of all performance metrics. For example, the number of measures that are improving versus the number of measures that are declining (screenshot to the left).

In comparison to some of the solutions that I have seen deployed, this one is pretty basic. That being said, the fact that the city has identified the performance measures that are critical for city services and has changed processes to capture these metrics, it is an important first step. To then "expose" this information to the general public in a live reporting system shows that they have committed to hold themselves accountable.

Based on reading the cryptic URLs while clicking through the different reports, it looks like CPR is using the Oracle Siebel Analytics tool. Just a guess.