Recommender Systems are the New Oil Refineries

From Crude to Shrewd

I read a post by a colleague of mine, Richard Kastelein, who covers the Social TV space in which he proclaims “Scene-Level Television Metadata – Tagging TV – is The New Oil in the Industry”.  There is a great deal of merit to this train of thought, and indeed having access to this kind of metadata about television programming has been a long time in coming.

I like his New Oil analogy and, given my interest in recommender systems, thought it appropriate to continue the analogy meme with the statement that Recommender Systems will be the New Oil Refineries.  

What does this mean?

Crude oil has limited applications until it as been processed and refined into myriad useful products.   Likewise, metadata itself provides a modicum of utility, but its full potential becomes realized when combined and processed with usage behavior. 

Until recently, comprehensive television usage data was not captured, as there was no pipeline to transport the information.  A burgeoning well of data was left untapped because of the limitations of the Set Top Box (STB) platform itself.  The STB had limited data bandwidth, processing power and the majority of its logic was burned into its firmware – making it inadaptable and expensive to upgrade in the field. 

Over The Top (OTT) solutions have arrived and are aimed squarely at solving the STB challenge.  These second screen companion devices (e.g. iPads, tablets) provide the facilities to capture quality data as well as the feedback mechanisms for retrieving that data in a timely fashion. 

OTT solutions will deliver the missing data pipeline and concomitant data elements that are the fuel needed to power sophisticated, data hungry recommendation platforms. 

What are the applications? 

Recommender systems have the capacity to analyze and decision against incredibly large datasets.  As noted above television content can be one of the recommended products delivered – but so can advertising; be it imbedded, pre or post roll.

Ready, Fire, Aim 

If it is indeed true that 82% of TV Ads generate negative ROI then surely it makes plain sense to have a powerful machine learning system in place that can accurately target content to users in real time. 

Television advertisers just don’t understand me 

The combination of new user interaction models, the availability of standardized TV metadata and powerful machine-learning infrastructures will deliver a more personalized experience to each user.  Indeed as users move en masse to tablets for companion viewing and in some instances primary viewing, or as systems allow for user logins, there will be an even greater opportunity to understand the individual.  This personalized experience will reach across all content – including advertising.

Not Just “New” Search

One of my favorite VC bloggers, Mark Suster, posted an article on his site not quite a year ago; The Future of Television & The Digital Living Room.  He touches on the subject of better search in in his 7^th point about Content Discovery and the improvement of the user experience.  What’s missing from his discussion is the introduction of recommendation engine services.  Better Search will not be the difference in next generation TV.  The power of understanding the consumer at a heretofore-unavailable scope via this new OTT data rich model of usage interaction combined with a wealth of television metadata is how we’ll be equipped to offer meaningful, relevant content discovery experiences.  

Consumers need relevant content served up to them - let’s fire up the refinery and make it happen.

Photo credits: © 2011 Eric Wilson

Just Say NOcial

Social Is The New Black

One of the most abused phrases I hear these days is Social + [insert any noun here].  Conventional wisdom appears to regard communication between people as a bold new concept.  This exciting new medium adds value to any business (check out the astronomical valuations of the privately held companies on SharesPost). 

Sarcasm aside -

so·cial (adjective) : offering opportunity for interaction; allowing people to meet and interact with others in a friendly way

The truth of the matter is that social is asked about whenever I discuss a recsys implementation with clients – no matter the vertical.  Some opportunities lend themselves well to an articulate social component plus a recsys while still others struggle to find social relativity.  Thus, I have proactively endeavored to find concrete strategies in which to use ‘social’ data in productive and meaningful ways, especially with respect to an implementation of a recsys.

Do I Really Want Recommendations From My Friends

We may have 500 friends on Facebook or Google Plus but do we really want each and every one of them driving recommendation decisions for us?  Imagine the experience you have every time you visit your wall, how it’s littered mostly with things that aren’t directly or immediately interesting to you.  Now imagine that same experience applied to music, books or TV.  Simply adding friends to the mix will not add value; more likely it will invoke churn from your service.

Hey friend, what do you like?

There are more informed approaches for incorporating the data from the social graph that can deliver a positive uptick in consumption.  One such approach is the representation of a user’s friend set in order to reinforce a recommendation.  This means we still rely on a user’s behavior within an ecosystem (supervised learning) to arrive at a recommendation decision but choose to complement the result with a list of friends that have also engaged with the recommended item.  In this way we benefit from both machine learning and the social graph – we just don’t use the social graph to drive the decision.  Cool.

Filter My Friends

Another interesting approach examines the relative usefulness (conversion percentage) of your friends’ recommendations to you.  Observing the success and failure of your friends’ recommendations to you effectively turns them into a channel from which statistical probabilities of success can be rendered.  

I like what you picked!

This success metric can be used to rank order future recommendations from ‘high quality’ friends and can cleanly winnow away the noise.

A beneficial side effect of using a qualified friend recommender is the possibility for more prolific experiences of serendipity.  The facility to surprise and delight a consumer cannot be underestimated.  While we are refining mathematical pathways to successfully discover interesting elements that fall outside the known model of the user, the incorporation of high quality external influencers is a welcome commodity.

Social DOES Have Value

When more than a hand wave toward social is given we can discover many influential uses for social data.  The overuse of the term will surely relax over time – I will just have to hold my tongue until the next big buzzword takes its place.

Photo credits: © 2011 Eric Wilson

Recommender System + OTT or STB = Personalized TV

What To Watch

There is so much good programmed content available today it has become a staggering proposition to sift through the morass and find things that are interesting, compelling or even stimulating.  This is both a good and bad thing for the end consumer.  The proliferation of content choices has made the position of consumer an arduous task – and search does not solve the challenge.  “We want to be entertained and don’t really want to work for it.”  It is this divide that has consumer businesses scrambling for a better mousetrap and more often than not these content megaliths are turning toward recommendation systems.

Understand Me Please

Consumers today are engaging with content and should be rewarded for it.  They are providing valuable insight into their interests simply by participating within the content ecosystem.  Each time a consumer sees a content element, they have the distinct opportunity to interact with it.  What they do and what they don’t do can shape the ongoing model of what that consumer should be presented with.  Time of day, device (i.e. tablet, mobile phone) or set top box provide more articulation of exposure and direction with respect to future interest probabilities.  Armed with user behavior, contents’ performance in a Darwinian ecosystem and meta-data, recsys solutions have a strong foundation from which to attack the burgeoning content discovery battle.

What will make this personalized TV movement fail, however, is by forcing the user to contribute meaningful information to the personalization model at every turn.  There is no way to realize the ‘lean back’ experience where content is magically presented in increasingly more accurate interest categories if the user has to proactively engage with the system.  This is where behavioral observation can play a lead role, allowing consumers to simply ‘be’ in the system.  The consumer can unassumingly go about her day by interacting with the things she finds appealing all the while a recsys is paying close attention and is learning about her interests and preferences in the background.

What about systems that allow consumers to rate things?  Cool – this is helpful information.  I would like to suggest that the star rating system and other incarnations like it be banned from use.  Consumers just don’t know what to do with stars in-between one and five.  The best rating approach is really ternary (i.e. thumbs-up, thumbs-down and neutral or no thumb).  It’s the clearest and most elegant way to allow consumers to express interest.  You can see this KISS principle in action at sites like YouTube.

“What about my privacy?”  This is one of the top questions that arise from discussions I have around recommendation system implementations.  My approach is always to let the consumer know up front what is trying to be accomplished by such an effort and to get the user excited and engaged with the experience.      Surreptitious gathering of user data has gotten some very large and well-known public companies into trouble.  There is no excuse for ‘spying’.  It’s not needed as long as the target goals have the consumer in mind.  Systems can be very successfully designed and implemented that do not require Personally Identifiable Information (PII).

The Road To Personalization

Keeping the aforementioned points in mind, the ideal of providing a compelling content experience is readily achievable.  Especially as a growing number of consumers employ the tablet as a second screen or even in some cases use it as the primary screen.  Terrific amounts of quality data can be gathered from these devices.  In turn this data will be digested by the recsys platforms, which will produce more targeted, relevant personalization experiences.

Personalization solutions will become table stakes in the new world of multi-device entertainment.  Those that realize this early will have a great advantage in the competitive marketplace of content and consumers.