Towards a curated web: why vertical search is a potential Google killer

May 24th, 2010 by Alex

In the last post in this series we looked at how vertical search sites run by affiliate marketers are changing the shape of the purchase funnel for goods and services online. We compared this “funnel 2.0″ to the old approach, in which surfers would try to zero in on the best available product and retailer through multiple general searches. At the end of the article we introduced the idea that this new funnel threatens the AdWords model which powers Google’s search business.

In this post we want to explore further Google’s uneasy relationship with vertical search, and in particular the extent to which vertical search poses a significant threat to the business model for general search. At Keplar we believe that there’s an opportunity emerging for a new entrant to massively disrupt search as it exists today, and I discuss this in the article as well.

Google’s attitude to vertical search is a complex one. On the one hand, we continue to hear rumours that Google will spend a billion dollars buying travel search engine ITA Software, which powers Kayak.com, Orbitz and others. On the other hand, Google regularly bans affiliates from advertising their vertical search sites using Google AdWords: complaints about these mass bannings can be found all across the Web.

What’s behind this difficult relationship with vertical search? To understand the reasons, we first need to understand the commercial dynamics of general search, and understand Google’s hardening commitments to that business model.

The commercial dynamics of general search

General search is a key navigation and discovery tool for the Web, and it has become increasingly dominant over time: growing numbers of users are abandoning browser bookmarks and the address bar altogether in favour of the power of the “Google command line”. But that utility comes at a massive cost: general search is dependent on Google (and of course Bing et al) crawling the entire known Web, storing and indexing all of that data and then returning it in any relevant search query.

These search results are costly to provide, and moreover most of this traffic is not monetisable. Whether I’m searching for help with a programming question, typing “bbc.co.uk” into the Google search box as a URL short-cut, or simply checking today’s weather – none of these will generate any revenue for Google.

The searches which are monetisable are the ones related to products and services which the user may want to buy online – for example, a t-shirt, or holiday flights, or a new car. The trouble though is that general search is a blunt tool for helping people make decisions about what to buy, especially compared to a vertical search engine which exists only to help people make decisions about a certain product category. So it’s no coincidence that if you search for “car insurance” on Google UK, the vast majority of the search results (both organic and paid-for) are for vertical search sites. Those sites are great at converting searchers into purchasers, they are well rewarded by merchants as a result, and thus are able to outbid (and out-link) competitors to come top of Google’s search results. We explored this “capability gap” between generalised and vertical search further in our last post.

So to recap: general search is expensive to offer, the majority of it is unmonetisable, and the piece that is monetisable (products and services) is more effectively monetised by vertical search engines run by domain experts. If this is the situation, what kind of threat does this pose for Google, and what opportunities does it provide for other players?

Google’s commitments to general search are hardening

Don Sull, a professor of management at London Business School, has developed a powerful theory to explain why large corporations so often get stuck in a rut and find themselves outmanouevered by competitors and new entrants. Don has identified what he calls “active inertia”, in which a company’s hardening commitments clash with turbulent markets and bring about its decline in three acts. His theory is laid out in an FT article which is well worth reading, but in summary the three acts are as follows:

  1. Act 1: Managers commit – the company’s management makes long-term commitments, such as public declarations around areas of focus, or investing in specialised resources
  2. Act 2: Commitments harden – these long-term commitments engender an attitude of “if it ain’t broke, don’t fix it”. The commitments establish a trajectory for the company, and all functions of the company start optimising themselves around this trajectory, to speed up execution against the long-term commitments
  3. Act 3: Active inertia – shifts in the external business environment render the company’s existing commitments outmoded or obsolete, and this changed environment stops rewarding progress along the established trajectory. The company typically responds by accelerating the activities that succeeded in the past, but this merely digs them deeper into the rut

It is the Keplar view that this “active inertia” theory applies as well to Google’s general search business, putting Google into the category of technology companies who maintain significant “legacy” platforms, including Microsoft (Windows, Office), eBay and, increasingly Apple (iPhone/AppStore). Make no mistake – these legacy platforms confer significant advantages on the incumbents (including significant revenue generating potential and partner lock-in), but they also significant restrict the flexibility to innovate in the face of changing business environments and disruptive innovations.

Google’s active inertia in search

So, Google has won the war of general search – but as we have suggested above, along with this victory comes a set of “hardening commitments” (to use Don Sull’s language) around supporting a hard-to-monetise utility search platform, even while affiliate marketers reshape the purchase funnel and capture significant value for themselves.

To date, Google’s response has been to build or buy search capabilities in key verticals (such as real estate, video and travel), but this approach appears to us to have two key weaknesses:

  1. Ignoring the long tail of verticals – there is a significant long tail of verticals out there, from cheese to vintage motorbikes to music festivals. However deep its pockets, Google cannot hope to provide vertical search functionality for all of these
  2. Preventing the survival of the fittest – Google’s AdWords model works because it creates a competitive environment in which the best advertisers win out (because they can afford to out-bid their rivals). Conversely by maintaining its own monoculture for vertical search – rather than letting multiple affiliates compete for each vertical – Google may ultimately deliver a poorer search experience for each niche. At the very least, it could create mistrust that Google will treat vertical search competitors fairly, as reflected in the Foundem case

In our view, Google’s current approach to vertical search reflects the “active inertia” problem identified by Don Sull: the changing search environment is no longer rewarding Google’s progress along its established trajectory, and Google’s build-and-buy response is merely exacerbating the problem.

A disruptive innovation: the aggregation of vertical search

If Google is suffering from “active inertia”, then what are its competitors doing? One potential source of disruptive competition could come in the form of the aggregation of vertical search. While the costs required to provide general web search are massive, the costs around aggregating a few thousand vertical search sites are comparatively tiny, especially if vertical search sites can be incentivised to conform to a certain standard (e.g. a semantic markup for vertical search results).

It is also not hard to envisage a competitive mechanism (e.g. real-time bidding) within an aggregator site which allows multiple vertical search sites to compete to provide a set of search results, ultimately driving up the quality of the aggregated search service in every niche. Such a platform would be hugely disruptive for Google’s existing search business.

Given the low costs to entry for an aggregator service such as this, we could see one being launched by a “tier two” general search provider such as Microsoft and Yahoo!, or even by a much smaller player with a strong pedigree in helping users find things online, such as Ask Jeeves.

If an aggregator like this is launched – and we can see no reason why one shouldn’t be – then its owner will have all of the benefits of accessing online shoppers at the time of strongest intention, but without all the costs associated with crawling, storing and indexing the whole Web. Think of it as a “curated web”: one where all the hassle of product discovery, selection and purchase is outsourced to enthusiasts who really “get” the niche that you are into.

Interested in improving your existing vertical search site, or developing a new one? Get in touch to find out how Keplar can help with your vertical search strategy, product design and technical implementation.

Leave a Reply