I was honored to present our Analytics Audit to a great audience at the Gateway Analytics Network here in San Francisco last week. As promised to those folks, I’m posting my slides. Don’t hesitate to give me a holler if you want to continue the discussion or learn more!

Analytics Roadmap

But in the meantime, I wanted to comment on a couple of articles I’ve seen in the last few weeks that have really got my brain engaged about the history of our sport, er, profession.  I recommend them all for reading and link following – there’s good stuff here.

First, Salon had an interview with Ann Blair from the Browser about the History of Information and how we evolved to collect, organize, and use data.  “Wow,” I thought, “there’s an official branch of history about what we do as data geeks!” I don’t know why that should surprise me, but there it is.  And while Blair points out that this field of study has grown significantly in the past decade, per her examples and book recommendations, in reality, “Data Science,” “Information Management,” and “Analytics” have been closely practiced for millennia.  This reminded me of the Silk Road examples that I used to use when discussing Retail Demand Analytics: merchants and traders have worried about their supply chains, inventory levels, and sales metrics since the days of the Silk Road.  So Walmart may have built the largest database, but the practice actually started a looong time ago.  BTW, Gil Press is working on a timeline of the specific history of Data Science and it looks great so far.  He’s asking for input on milestones, so help out if you can!

With all of that rolling around in my head, I came to Bob Warfield’s post on Big Data = BI + ADD.  In general, I completely agree with him that we’ve been dealing with BD for a while.  But there are a couple of things that feel different to me after 20+ years of doing this:

• It’s not just about volume anymore.  Yes, there’s a lot of data in daily level retail sales for 100’s of products across storefronts from San Antonio or Sydney.  But the thing that always killed us were the different types of data.  And today it’s not daily, it’s hourly or transactional, minute by minute, second by second, nano by nano…you get the picture, things are changing.
• It’s not just retail and financial services anymore.  All sorts of industries are realizing that they can’t be competitive without harnessing the volume of all the different stuff they call data.
• Yes, we’ve always has some technology and technologists that could deal with lots of data and complex questions (Bob shows a perfect example in this post about his work for eBay), but those solutions weren’t feasible for the majority of companies or ordinary analysts.  That starting to change.

But the last point that Warfield makes is spot on:  It’s not about the data, which is important, but it’s about the questions.  And who is asking them?  And how good are they (we) at asking them? That’s where we really need to spend the time and energy.  Not on “can the database crunch the 0’s and 1’s” but on: “Do the right people who know their business best have the wherewithal to ask and answer their Right Questions?”  And do those people have access to the right tools to get their Big Answers?  Don’t get me wrong—data scientists are important and worth their weight in gold.  But so are the sales managers and shop floor leaders who really know what’s important for their customers’ success.

I like his idea – Big Data really should be Big Questions.  Or, rather, Big Answers.

Forward we go!

During the first dotcom boost (boom/bust), I actually kept a log of my commute times.  Yes, I am geek and proud, so there!  From 1998 to 2001, my commute from Foster City to Mountain View increased and then decreased by 10 mins.  Unfortunately, that notebook is long lost, because I’d love to match it up against the analysis that I just completed using the PatternBuilders FinServ product FinancePBI.

Since I had lost the log I decided to practice some Google Fu and see if there was public data available that could be used to do the analysis.  The closest thing was this amazing trove of road sensor data from pems.dot.ca.gov .  Wow, what a treasure trove!  I’ve loaded 3.5 years of daily “flow” data (count of cars) for the sensors that stretch from Gilroy to Marin on 101, 280, 580, 680, etc. – basically all the major highways in the Bay Area, about 2M geocoded sensor reads that when loaded up gives us about 6 MM data points when you include meta data.  Given how lusciously detailed and granular the data is, I knew I would never get anywhere with just Excel.  But as I mentioned I have a friend in big data, so I just loaded the data up into the PBI cloud using the csv import feature.

For an “economic” factor, I’m currently using the Stock Ticker data from the PBI Finance solution. The instance I was using had over 10 years of data from now until 2002. (See below for some other ideas I’m chewing on.)  So using flow across a couple of sensors in Mountain View, I’ve matched it up against the Stock Ticker Data for two of my favorite companies – Intuit and Google – that both have large campuses nearby.

What did I find? Before I tell you, first let’s take a look at how I used PatternBuilders to find it.  My first step was to look at the sensors on the Map view.  Even though I’ve been driving over them for years, it’s not like there are signposts labeling each sensor, so I needed to find the sensors that employees of those companies would drive over to get to and from work.  Since the data from the state came with latitude & longitude data for each sensor and the PBI import utility automatically geocodes imported data with geographical information, it was easy for me to find sensors that tracked my old commute to the area using PBI’s map view.

Once I knew what sensors to look at, I took a look at the data for closing price vs. daily flow at those sensors just to get a feel for the raw data and look for outliers (failed sensor being the most likely cause) and other factors that might affect the results of a Pearson’s correlation. Here’s a couple of views from the Line Chart view that shows the closing price for INTU and GOOG as well as the daily flow for the sensors.

Intuit’s stock consistently did well and Google showed quite a bit of volatility.  Another thing to notice is the outlier readings from sensor 401360 in 2010.  Marilyn’s first law of data geekdom – if you don’t take a look at the raw data first, before you start crunching and chewing, it will give you indigestion.  You never know what’s hiding in some random week some random when.

A quick look at the above curves shows that there might be something there…but I need to run a real correlation.  Luckily all that takes is a click on the correlation menu and ….Voila!  Correlation!  Which Is Not Causation.  I won’t say that again, but it should be remembered (and will be by the conscientious among you) as you read the rest of this post.

(BTW, this screen shot shows the data grid that drives the charts in PB.  It can be hidden as I did in the above pics.)

So what do we see?  Well, Intuit’s stock price seems to be more closely related to changes in the traffic flow around its campus than Google’s.  The hovering zoom in the graphic shows the line graph for the two data points beneath the correlation – in the case of the above its INTU versus sensor 401360.  (Also, you’ll notice that I changed the dates using the cool date UI slider at the top of the chart to take out the outliers.  If you’re curious, the correlations are more extreme with those data points included.)  The view below shows the details behind the negative correlation for Google and sensor 402378.

And the meaning here?  Well I’d love your thoughts, but one hypothesis we like is that since Google is a much more “global” company, its closing price is much less connected to activities at its Peninsula office.  Intuit on the other hand is a very centralized company compared to Google with most of its core functions (software development, sales, etc.) centered at its Mountain View location – which means that you would expect that if there is a correlation between traffic to a company and a company’s performance that the correlation would be stronger.  For this time period for Intuit and Google the data we have certainly supports that hypothesis.  There are a lot of confounding factors here, most obviously the greater volatility of Google vs. Intuit.  But certainly this relationship looks like a good research candidate for folks who trade or analyze stocks for a living!

What’s Next?

I’m not done here…and I’ll take any and all ideas you guys have.  I’m digging around for other economic factors – like unemployment, jobless claims, business starts, real estate changes, etc. – to match up against the road data.  You have ideas?

Yes, they were concerned that they were missing the peak selling times during the day when consumers came looking for their produce.  They had enough signals to optimize their supply chain on a daily basis, but they didn’t know if their sections were being stocked during the day at the right intervals.  Just because their fresh produce was in the backroom, didn’t mean it was going to sell!

All that wonderful RFID data (about 12M reads for 6 months) added to the hourly POS data from the Retailer could give us a clue as to what was happening.  Now dealing with RFID data is messy, to say the least.  There is sometimes more noise than music, as I like to say.  But we cleaned it up (threw out the reads that had a case of the Produce’s product going in and out of the stockroom 13000 times in one week), and compared the intra-day RFID movements from the stockroom to the floor against the hourly POS sales.

And indeed, most stocking of their product – based on the first read of an RFID tag moving to the stockroom – happened between 7 and 9am.  And their sales peak – well it started around 11a and increased until 6 p or so.  There was definitely a risk that around 3 or 4, the product on the shelf wasn’t the best or freshest the store had to offer.

There was also some evidence that the days that suffered the most from out-of-stocks (defined by a day when the stock fell below a goal) had the worst re-stocking patterns.  Produce didn’t have access to intra-day inventory levels, but we calculated as estimate based on the end of day inventory and the RFID stocking movements.  Looking at the peak selling times and putting OOO into quartile buckets, it became clear that the when re-stocking happens during the peak selling time, there are fewer OOOs.

Now, this may all sound really obvious.  But here’s the kicker.  Produce had held this opinion (hypothesis) for a long time and had argued with Retailer for months about the restocking practice.  But Retailer refused to change their personnel practice…until they saw the data.  That opened the door for a lot of productive conversations between Produce and Retailer.