In the last post in this series we looked at how vertical search sites run by affiliate marketers are changing the shape of the purchase funnel for goods and services online. We compared this “funnel 2.0″ to the old approach, in which surfers would try to zero in on the best available product and retailer through multiple general searches. At the end of the article we introduced the idea that this new funnel threatens the AdWords model which powers Google’s search business.

In this post we want to explore further Google’s uneasy relationship with vertical search, and in particular the extent to which vertical search poses a significant threat to the business model for general search. At Keplar we believe that there’s an opportunity emerging for a new entrant to massively disrupt search as it exists today, and I discuss this in the article as well.

Google’s attitude to vertical search is a complex one. On the one hand, we continue to hear rumours that Google will spend a billion dollars buying travel search engine ITA Software, which powers Kayak.com, Orbitz and others. On the other hand, Google regularly bans affiliates from advertising their vertical search sites using Google AdWords: complaints about these mass bannings can be found all across the Web.

What’s behind this difficult relationship with vertical search? To understand the reasons, we first need to understand the commercial dynamics of general search, and understand Google’s hardening commitments to that business model.

The commercial dynamics of general search

General search is a key navigation and discovery tool for the Web, and it has become increasingly dominant over time: growing numbers of users are abandoning browser bookmarks and the address bar altogether in favour of the power of the “Google command line”. But that utility comes at a massive cost: general search is dependent on Google (and of course Bing et al) crawling the entire known Web, storing and indexing all of that data and then returning it in any relevant search query.

These search results are costly to provide, and moreover most of this traffic is not monetisable. Whether I’m searching for help with a programming question, typing “bbc.co.uk” into the Google search box as a URL short-cut, or simply checking today’s weather – none of these will generate any revenue for Google.

The searches which are monetisable are the ones related to products and services which the user may want to buy online – for example, a t-shirt, or holiday flights, or a new car. The trouble though is that general search is a blunt tool for helping people make decisions about what to buy, especially compared to a vertical search engine which exists only to help people make decisions about a certain product category. So it’s no coincidence that if you search for “car insurance” on Google UK, the vast majority of the search results (both organic and paid-for) are for vertical search sites. Those sites are great at converting searchers into purchasers, they are well rewarded by merchants as a result, and thus are able to outbid (and out-link) competitors to come top of Google’s search results. We explored this “capability gap” between generalised and vertical search further in our last post.

So to recap: general search is expensive to offer, the majority of it is unmonetisable, and the piece that is monetisable (products and services) is more effectively monetised by vertical search engines run by domain experts. If this is the situation, what kind of threat does this pose for Google, and what opportunities does it provide for other players?

Google’s commitments to general search are hardening

Don Sull, a professor of management at London Business School, has developed a powerful theory to explain why large corporations so often get stuck in a rut and find themselves outmanouevered by competitors and new entrants. Don has identified what he calls “active inertia”, in which a company’s hardening commitments clash with turbulent markets and bring about its decline in three acts. His theory is laid out in an FT article which is well worth reading, but in summary the three acts are as follows:

1. Act 1: Managers commit – the company’s management makes long-term commitments, such as public declarations around areas of focus, or investing in specialised resources
2. Act 2: Commitments harden – these long-term commitments engender an attitude of “if it ain’t broke, don’t fix it”. The commitments establish a trajectory for the company, and all functions of the company start optimising themselves around this trajectory, to speed up execution against the long-term commitments
3. Act 3: Active inertia – shifts in the external business environment render the company’s existing commitments outmoded or obsolete, and this changed environment stops rewarding progress along the established trajectory. The company typically responds by accelerating the activities that succeeded in the past, but this merely digs them deeper into the rut

It is the Keplar view that this “active inertia” theory applies as well to Google’s general search business, putting Google into the category of technology companies who maintain significant “legacy” platforms, including Microsoft (Windows, Office), eBay and, increasingly Apple (iPhone/AppStore). Make no mistake – these legacy platforms confer significant advantages on the incumbents (including significant revenue generating potential and partner lock-in), but they also significant restrict the flexibility to innovate in the face of changing business environments and disruptive innovations.

Google’s active inertia in search

So, Google has won the war of general search – but as we have suggested above, along with this victory comes a set of “hardening commitments” (to use Don Sull’s language) around supporting a hard-to-monetise utility search platform, even while affiliate marketers reshape the purchase funnel and capture significant value for themselves.

To date, Google’s response has been to build or buy search capabilities in key verticals (such as real estate, video and travel), but this approach appears to us to have two key weaknesses:

1. Ignoring the long tail of verticals – there is a significant long tail of verticals out there, from cheese to vintage motorbikes to music festivals. However deep its pockets, Google cannot hope to provide vertical search functionality for all of these
2. Preventing the survival of the fittest – Google’s AdWords model works because it creates a competitive environment in which the best advertisers win out (because they can afford to out-bid their rivals). Conversely by maintaining its own monoculture for vertical search – rather than letting multiple affiliates compete for each vertical – Google may ultimately deliver a poorer search experience for each niche. At the very least, it could create mistrust that Google will treat vertical search competitors fairly, as reflected in the Foundem case

In our view, Google’s current approach to vertical search reflects the “active inertia” problem identified by Don Sull: the changing search environment is no longer rewarding Google’s progress along its established trajectory, and Google’s build-and-buy response is merely exacerbating the problem.

A disruptive innovation: the aggregation of vertical search

If Google is suffering from “active inertia”, then what are its competitors doing? One potential source of disruptive competition could come in the form of the aggregation of vertical search. While the costs required to provide general web search are massive, the costs around aggregating a few thousand vertical search sites are comparatively tiny, especially if vertical search sites can be incentivised to conform to a certain standard (e.g. a semantic markup for vertical search results).

It is also not hard to envisage a competitive mechanism (e.g. real-time bidding) within an aggregator site which allows multiple vertical search sites to compete to provide a set of search results, ultimately driving up the quality of the aggregated search service in every niche. Such a platform would be hugely disruptive for Google’s existing search business.

Given the low costs to entry for an aggregator service such as this, we could see one being launched by a “tier two” general search provider such as Microsoft and Yahoo!, or even by a much smaller player with a strong pedigree in helping users find things online, such as Ask Jeeves.

If an aggregator like this is launched – and we can see no reason why one shouldn’t be – then its owner will have all of the benefits of accessing online shoppers at the time of strongest intention, but without all the costs associated with crawling, storing and indexing the whole Web. Think of it as a “curated web”: one where all the hassle of product discovery, selection and purchase is outsourced to enthusiasts who really “get” the niche that you are into.

Interested in improving your existing vertical search site, or developing a new one? Get in touch to find out how Keplar can help with your vertical search strategy, product design and technical implementation.

Anyone watching TV these days can’t help but notice the proliferation of advertisements for price comparison sites. What is less widely understood is what is driving this sudden growth, and how this is changing the customer purchase funnel (in other words, the way we select products and services to buy online).

In our previous blog post in the Curated Web series, we looked at the various types of vertical search site, how they make money and where they get their search results from. We touched briefly on affiliate marketers, mentioning the affiliate networks which they belong to and the affiliate fees which they earn. In this post we dive much more deeply into affiliate marketing, to understand why retailers are spending increasing amounts on affiliate marketing, what impact these rapidly developing vertical search sites are having on buyer behaviour online, and what these changes mean for major Web companies, especially Google. But first, we will explain a little more about what affiliate marketing actually is.

A quick primer on affiliate marketing

Affiliate marketing is the Web’s second oldest profession. In a nutshell, retailers offer “affiliates” a fee (either a flat fee or a percentage) based on the number of customers that an affiliate sends them and the amount that those customers spend. Anyone who can somehow find prospective customers and direct them to a retailer’s site can be an affiliate, and there are legions of affiliates doing exactly this – building assorted price comparison sites, product comparison sites, review sites, voucher sites and directories. For the retailer, affiliate marketing offers them a way of outsourcing some of their marketing effort. For clever affiliates who are good at finding ways to attract and direct prospective customers, it offers a potentially lucrative revenue stream.

One way in which affiliates can make money is by setting up vertical search sites for particular products or services. A good vertical search site guides the individual into making a buying choice, by:

1. Identifying the key factors that matter to that individual,
2. presenting him or her with the options available,
3. offering expert advice on the pros and cons of different options,
4. offering user opinions (via social media) from other people who’ve made that decision,
5. and then finally providing a straightforward interface so that the individual can make a buying decision and then go quickly on to make the purchase.

Two examples of sites that do this particularly well are Money Supermarket, especially for financial products (e.g. credit cards, mortgages) and Simplify Digital, for people looking to choose a TV / broadband deal. These are effective affiliate sites because they add value for prospective customers (by helping them make informed buying decisions) whilst adding value to retailers by providing them “pre-qualified” customers who’ve expressed an interest in their specific product / service. Popularity with prospective customers means that these sites attract a good volume of traffic, lowering the marketing cost they have to spend to attract users and increasing the profit they make (by effectively “selling on” the individual to a retailer for a higher fee).

How affiliate marketing is drastically altering the customer purchase funnel online

Affiliate sites, not least those with rich vertical search functionality, are having a significant impact on the way in which people make buying decisions online. The nature of this change is best summed up by a pair of “before and after” diagrams, looking at how the purchase funnel for buying (for example) home broadband has been altered by the advent of affiliate sites. The first diagram shows the cyclical set of general search querying and site browsing which consumers depended on before the advent of vertical search sites:

In the “after” diagram below, we show how a vertical search site takes over from generalised search to provide the end-to-end purchase funnel for customers:

From the diagrams above, two things are clear:

1. A good vertical search site simplifies life for the online shopper
2. A good vertical search site diminishes the amount of time that shopper spends doing generalised search, and indeed the importance of those search results

The future growth outlook for vertical search sites is extremely positive

The affiliate market continues to grow – retailers are allocating bigger portions of their marketing budgets to affiliate marketing, as it represents a very low risk way of growing sales (because they only have to pay when they make sales). At the same time, more and more affiliates are being attracted into the space by the growing number of people who are researching buying decisions online and / or going on to make those purchases online.

Vertical search sites are especially common in product categories where the value of a prospective customer is high – for example, mortgages, pay-TV, mobile phones; in these cases, retailers can afford to offer high acquisition fees to affiliates, which in turn attracts more affiliates to the space. But as competition heats up among affiliates for the high-value sales, we can expect to see more affiliates launching vertical search sites to cover the “long tail” of niche or low-value product categories, especially as the costs of setting up vertical search sites diminish.

So what next for general search?

To date, generalised search as offered by Google and others has been an extremely valuable service, because it offers advertisers (retailers) a way to reach prospective consumers looking for their specific products. Google’s AdWords service, where advertisers bid to reach users searching for particular terms, has long been that company’s cash cow.

With the growth in vertical search, however, advertisers are being offered the opportunity to buy prospective customers in places other than on Google search results. Furthermore, they may be able to better qualify those prospective customers, because a good vertical search site “walks” the customer through the buying decision funnel, and only passes them onto the retailer once they’ve made a decision, providing a potentially higher value source of website visitors than AdWords. On top of this, some of these vertical search sites are now disintermediating Google, appealing directly to shoppers via TV advertisements and other above-the-line promotions.

So whilst the growth in vertical search may be good for both consumers and retailers, it could well spell trouble for Google – and we will explore this further in the next post in this series.

As the Google competition issues rumble on, various commentators have been grappling with the concept of general versus vertical search. Here at Keplar we have some thoughts on the rise of vertical search and its implications for Google, but we are aware that many people out there don’t yet know what vertical search is, or why it’s important. In this blog post we aim to nail down exactly what vertical search is, so that we can comment more widely on the emerging trends in future posts.

To begin with a definition: unlike a general Web search engine like Google or Bing, a vertical search engine focuses on a specific segment of content. Users visit these vertical search sites to conduct a specialised search for a specific genre of content or category of product. The most commonly known vertical search engines are the price comparison sites, like Kayak or Moneysupermarket.com, where consumers can enter their specific requirements and find a holiday or an insurance deal or similar.

But vertical search is not just used to drive e-commerce sales. There are vertical search engines for images (Google Images), jobs (Trovit Jobs) and dates (Single View) – to paraphrase a certain ad, there’s a vertical search site for pretty much everything.

Many vertical search sites are extremely cash generative. These revenues come largely through affiliate fees – if a user clicks through from Confused.com to Virgin Money and buys travel insurance, then Confused.com makes a referral fee. However there are others which are run at a loss (like Google Images) or as a labour of love; there are also many which don’t make money from the search results but monetize their regular audience through advertising – e.g. Scubadviser.

So vertical search engines are monetizable, but where do they get the actual search results from? There are four different approaches to collecting this data:

1. Pull: scraping – where a vertical search site builds its search index by ‘scraping’ relevant web sites. Scraping is the process of automatically collecting Web information and turning it from unstructured, human-readable data into structured data – perfect for feeding into a vertical search engine. (We wrote a previous blog post on this.)
2. Pull: affiliate networks – where a site imports its search index wholesale from one or more affiliate networks such as TradeDoubler or Commission Junction. These networks hold large databases of products from the merchants they represent – for example the Lastminute.com account within Commission Junction currently has 91 holidays in its database. A vertical search site for holidays could add all of these products to its search index.
3. Push: feeds – where a site builds its search index from merchant-submitted data feeds, typically in XML format. Good examples of this are Kelkoo and Google Product Search.
4. Push-pull: manual – where a vertical search site manually builds up a list of links to relevant sources, through a mixture of techniques including general web searches, community contributions and face-to-face meetings.

If we now understand what a vertical search site is, then how do these fit alongside general web search from Google or Bing? Are they complementary, or competitive? We’ll be discussing these questions and others in our next post in the Curated Web series…

This is the first post in our Curated Web series of posts. The second post looks at how affiliate marketers have re-shaped the purchase funnel online, while the third and final post in the series looks at how these developments threaten Google.

How is web scraping useful for a business that isn’t Google-sized?  Web scraping can be used to collect and structure competitor data, making it an incredibly powerful marketing intelligence tool.  Consider online retail: using a web scraper it is possible for a retailer to automatically survey the range of products offered by competitor sites and the price each product is offered at. Because web scrapers can be automated, they can be programmed to run regularly – so companies can analyse how sensitive their own sales volumes are not just as a function of the item price, but as a function of the prices competitors price them at. It is even possible to use the data from web scrapers to dynamically price items in an online shop so that they are always competitive. If you’re an online retailer, you are quite possibly already being regularly scraped by a competitor.

In this post, we provide an example of a simple web scraper built using Groovy. We chose Groovy because it’s a powerful, agile scripting language which is great at navigating/analysing HTML. The target of our scraper is a simple test “shop” which we have setup in Keplar Labs. You are welcome to run this scraper against our test shop – please note that scraping other sites may be against their terms and conditions or even in some cases an offence. Please seek legal advice before running any scraper on someone else’s website.

Without further ado, here is the Groovy code:

/* Intelbot written by Alex Dean on 15 Jan 2010 with dependencies:
 *  - NekoHTML parser (latest stable), http://nekohtml.sourceforge.net/index.html
 *  - Xerces (2.0.0 or higher), http://www.apache.org/dist/xerces/j/
 */

// Define the pages which contain links to products - our "seeds" in crawl parlance.
def seeds = ["http://labs.keplarllp.com/shop"]

// Load the NekoHTML parser with Xerces - this lets us parse the HTML.
slurper = new XmlSlurper(new org.cyberneko.html.parsers.SAXParser())

// Now let's loop through each seed URL in turn.
seeds.each() {

	println "Accessing seed URL ${it}"
	def seedURL = new URL(it)

	seedURL.withReader { seedReader ->
		def seedHTML = slurper.parse(seedReader)

		// Show the title of the seed page we're parsing.
		println "Seed page title is ${seedHTML.depthFirst().grep{ it.name() == 'TITLE'}}"

		// Now loop through and find all the product links on this page.
		// For our purposes, a product link is any A tag inside a box div on the page.
		def productLinks = seedHTML.depthFirst().grep{ it.name() == 'DIV' && it.@class == 'box' }.collect { it.A.'@href'.toString() }
		productLinks.each {

			println "  Accessing product URL ${it}"
			def productURL = new URI(seedURL.toString()).resolve(it).toURL()

			productURL.withReader { productReader ->
				def productHTML = slurper.parse(productReader)

				// Now display the product name.
				println "    Name is ${productHTML.depthFirst().grep{ it.name() == 'H1'}}"

				// Now display the product description.
				println "    Description is ${productHTML.depthFirst().grep{ it.name() == 'P' && it.@class == 'ProductDescription'}}"

				// Now need to grab the product price.
				println "    Price is ${productHTML.depthFirst().grep{ it.name() == 'P' && it.@class == 'ProductPrice'}}"
			}
		}
	}
}

The code should be fairly self-documenting: essentially there is a loop to process each “seed page” (we have only one such page), and then an inner loop to process each product page found on a given seed page.

When you run the script in the Groovy Console, you should see output like this:

Accessing seed URL http://labs.keplarllp.com/shop
Seed page title is [Keplar Labs]
  Accessing product URL http://labs.keplarllp.com/shop/product1.html
    Name is [Flight to Dubai]
    Description is [Economy class flight to Dubai with Emirates]
    Price is [£699.00]
  Accessing product URL http://labs.keplarllp.com/shop/product2.html
    Name is [Train to Paris]
    Description is [First class train ticket to Paris on Eurostar]
    Price is [£225.50]
  Accessing product URL http://labs.keplarllp.com/shop/product3.html
    Name is [Flight to New York]
    Description is [Economy class flight to New York on United]
    Price is [£375.00]

As you can see, the scraper has successfully identified three products linked from the seed page, then accessed each of these product pages and retrieved the key information about each product (its name, description and current price). A more sophisticated scraper would simply build on this functionality, for example adding error handling and potentially adding some caching.

Web scraping is an incredibly powerful tool that has already transformed the web (by enabling services like Google search) but we believe it has the power to transform regular businesses too – once they learn to employ it. It’s also cheap to perform, making it a cost effective tool. So do try out the above code on your own computer – and let me know how you get on in the comments.

Interested in having a web scraper designed and built for your company? Send us an email to find out how Keplar can help.

Yesterday’s Financial Times reported that Google intends to launch an online property portal in the UK, where estate agents will be able to advertise their properties for free. “Experts say this could pose a serious threat to existing property websites and local newspapers”, according to the FT.

What this says about the future of search

The interesting thing about the announcement is not what it says about the future of the selling and renting property in the UK. It’s about the importance of vertical search relative to the generalised search market Google already dominates. By providing a specialised solution for people searching for property, Google is acknowledging that a vertical-specific search approach has advantages over a generalised product, and that ultimately there may be much more money in specific verticals than in general search. (Just think about the potential affiliate fees a property seller might be willing to pay to sell a half-million pound property.) We will explore the growing value in vertical search in more detail in a future blog post.

Long term, Google’s move is much more of a threat to estate agents than local newspapers and property sites

As anyone who’s recently tried to buy or rent a property in the UK’s supply-constrained market will testify, the UK’s “big three” property aggregation sites (Rightmove, FindaProperty and Globrix) already have good-enough functionality – in fact they already do all of the things that have excited commentators about Google’s offering, for example showing available properties on a map.

The problem for anyone relying on these sites is not one of  functionality, but one of recency: the property market moves quickly, especially in these supply-constrained times, and the stock available on these aggregation sites is rarely up-to-date. Remember that estate agents all have bricks-and-mortar outlets and a steady stream of new clients through the door – and they are also very effective at client retention. The upshot of all this is that they have little incentive to keep their online sites up-to-date with the latest properties. As a result, the properties that are advertised online – and still available – tend to be the runt of the litter. You could say that estate agents use sites like Rightmove in the way that online publishers use remnant ad networks: for the crap inventory which they can’t sell directly.

Towards a self-serve property market

Now, there is no reason to think that estate agents will be any faster uploading properties to Google’s site than to Rightmove. But, there is reason to think Google will do a better job at directing large numbers of interested web surfers to their site – no one is better at directing web traffic than Google.

And the other thing which Google excel at is opening up their platforms to smaller players, offering easy-to-use, competitively-priced self-serve interfaces. And if I want to sell my property, wouldn’t I prefer to do it directly on Google, and pay 0.02% commission, instead of with an estate agent where I pay 2%, so that they can advertise it on Google for free?