Towards a curated web: why vertical search is a potential Google killer

May 24th, 2010 by Alex

In the last post in this series we looked at how vertical search sites run by affiliate marketers are changing the shape of the purchase funnel for goods and services online. We compared this “funnel 2.0″ to the old approach, in which surfers would try to zero in on the best available product and retailer through multiple general searches. At the end of the article we introduced the idea that this new funnel threatens the AdWords model which powers Google’s search business.

In this post we want to explore further Google’s uneasy relationship with vertical search, and in particular the extent to which vertical search poses a significant threat to the business model for general search. At Keplar we believe that there’s an opportunity emerging for a new entrant to massively disrupt search as it exists today, and I discuss this in the article as well.

Google’s attitude to vertical search is a complex one. On the one hand, we continue to hear rumours that Google will spend a billion dollars buying travel search engine ITA Software, which powers Kayak.com, Orbitz and others. On the other hand, Google regularly bans affiliates from advertising their vertical search sites using Google AdWords: complaints about these mass bannings can be found all across the Web.

What’s behind this difficult relationship with vertical search? To understand the reasons, we first need to understand the commercial dynamics of general search, and understand Google’s hardening commitments to that business model.

Read the rest of this entry »

Towards a curated web: affiliate marketing and the new shape of the customer purchase funnel

May 19th, 2010 by Yali

Giant termite mound and elephant

Anyone watching TV these days can’t help but notice the proliferation of advertisements for price comparison sites. What is less widely understood is what is driving this sudden growth, and how this is changing the customer purchase funnel (in other words, the way we select products and services to buy online).

In our previous blog post in the Curated Web series, we looked at the various types of vertical search site, how they make money and where they get their search results from. We touched briefly on affiliate marketers, mentioning the affiliate networks which they belong to and the affiliate fees which they earn. In this post we dive much more deeply into affiliate marketing, to understand why retailers are spending increasing amounts on affiliate marketing, what impact these rapidly developing vertical search sites are having on buyer behaviour online, and what these changes mean for major Web companies, especially Google. But first, we will explain a little more about what affiliate marketing actually is.

Read the rest of this entry »

Towards a curated web: a quick primer on vertical search

April 1st, 2010 by Alex

As the Google competition issues rumble on, various commentators have been grappling with the concept of general versus vertical search. Here at Keplar we have some thoughts on the rise of vertical search and its implications for Google, but we are aware that many people out there don’t yet know what vertical search is, or why it’s important. In this blog post we aim to nail down exactly what vertical search is, so that we can comment more widely on the emerging trends in future posts.

To begin with a definition: unlike a general Web search engine like Google or Bing, a vertical search engine focuses on a specific segment of content. Users visit these vertical search sites to conduct a specialised search for a specific genre of content or category of product. The most commonly known vertical search engines are the price comparison sites, like Kayak or Moneysupermarket.com, where consumers can enter their specific requirements and find a holiday or an insurance deal or similar.

Read the rest of this entry »

Better competitive intelligence through scraping with Groovy

January 20th, 2010 by Alex

For the first of our series of technical posts I’m going to look at the poorly understood topic of web scraping. To start with a definition: web scraping is the process of automatically collecting Web information and turning it from unstructured, human-readable data into structured data that can be stored and analysed in a database or spreadsheet. The most famous scraper of all is Google, who regularly scrape and index a huge proportion of the Web to feed into their search engine.

How is web scraping useful for a business that isn’t Google-sized?  Web scraping can be used to collect and structure competitor data, making it an incredibly powerful marketing intelligence tool.  Consider online retail: using a web scraper it is possible for a retailer to automatically survey the range of products offered by competitor sites and the price each product is offered at. Because web scrapers can be automated, they can be programmed to run regularly – so companies can analyse how sensitive their own sales volumes are not just as a function of the item price, but as a function of the prices competitors price them at. It is even possible to use the data from web scrapers to dynamically price items in an online shop so that they are always competitive. If you’re an online retailer, you are quite possibly already being regularly scraped by a competitor.

In this post, we provide an example of a simple web scraper built using Groovy. We chose Groovy because it’s a powerful, agile scripting language which is great at navigating/analysing HTML. The target of our scraper is a simple test “shop” which we have setup in Keplar Labs. You are welcome to run this scraper against our test shop – please note that scraping other sites may be against their terms and conditions or even in some cases an offence. Please seek legal advice before running any scraper on someone else’s website.

Without further ado, here is the Groovy code:

Read the rest of this entry »

Estate agents (not local newspapers) should worry about Google’s entry into the UK property market

December 4th, 2009 by Yali

Yesterday’s Financial Times reported that Google intends to launch an online property portal in the UK, where estate agents will be able to advertise their properties for free. “Experts say this could pose a serious threat to existing property websites and local newspapers”, according to the FT.

What this says about the future of search

The interesting thing about the announcement is not what it says about the future of the selling and renting property in the UK. It’s about the importance of vertical search relative to the generalised search market Google already dominates. By providing a specialised solution for people searching for property, Google is acknowledging that a vertical-specific search approach has advantages over a generalised product, and that ultimately there may be much more money in specific verticals than in general search. (Just think about the potential affiliate fees a property seller might be willing to pay to sell a half-million pound property.) We will explore the growing value in vertical search in more detail in a future blog post.

Long term, Google’s move is much more of a threat to estate agents than local newspapers and property sites

As anyone who’s recently tried to buy or rent a property in the UK’s supply-constrained market will testify, the UK’s “big three” property aggregation sites (Rightmove, FindaProperty and Globrix) already have good-enough functionality – in fact they already do all of the things that have excited commentators about Google’s offering, for example showing available properties on a map.

The problem for anyone relying on these sites is not one of  functionality, but one of recency: the property market moves quickly, especially in these supply-constrained times, and the stock available on these aggregation sites is rarely up-to-date. Remember that estate agents all have bricks-and-mortar outlets and a steady stream of new clients through the door – and they are also very effective at client retention. The upshot of all this is that they have little incentive to keep their online sites up-to-date with the latest properties. As a result, the properties that are advertised online – and still available – tend to be the runt of the litter. You could say that estate agents use sites like Rightmove in the way that online publishers use remnant ad networks: for the crap inventory which they can’t sell directly.

Towards a self-serve property market

Now, there is no reason to think that estate agents will be any faster uploading properties to Google’s site than to Rightmove. But, there is reason to think Google will do a better job at directing large numbers of interested web surfers to their site – no one is better at directing web traffic than Google.

And the other thing which Google excel at is opening up their platforms to smaller players, offering easy-to-use, competitively-priced self-serve interfaces. And if I want to sell my property, wouldn’t I prefer to do it directly on Google, and pay 0.02% commission, instead of with an estate agent where I pay 2%, so that they can advertise it on Google for free?