Amazon’s Mechanical Turk provides a very convenient platform for getting large numbers of workers to perform manual steps as part of large scale business processes, such as cleaning data sets for use in machine-learning algorithms, or moderating content.

However, it is not enough for Mechanical Turk to provide results fast. The results themselves need to be reliable and hence it is critical that companies using Mechanical Turk invest in a suitable strategy for accuracy.

The mirror in the Hubble Space Telescope, the most precise ever made, was initially 10 nanometers off the correct curvature. The inaccuracy was catastrophic and cost several million dollars to fix

Amazon provides two primary tools for helping users validate the accuracy of results. We’ll look at these both briefly, before outlining a third technique which, used in combination with the first two, can be used to deliver a very rigorous approach to accuracy. These three strategies for accuracy are as follows:

1. Use multiple workers to perform each task independently and compare the results

Mechanical Turk is designed to allow the same task to be given to multiple different workers, and makes it easy to compare their different responses.  It is possible, then, to accept all results where there is consensus amongst the different workers, and to manually check (or even just disregard) results when there is a discrepancy.

The trouble with this approach is that even when all workers give the same answer, it is still possible that they all wrong. In the latter half of this post, we will show how you can begin to quantify that probability, and hence start to accurately measure the confidence levels of your results.

2. Make workers perform a set of qualifying tasks

Amazon makes it very easy to define your own qualifications, assign them to some of your workers (based, presumably on that worker’s accuracy on previous tasks) and then allow only “qualified” workers to complete future tasks.

This functionality makes it straightforward for companies to set pre-task “tests” where the correct answers are known, and workers’ answers can then be compared against the known answers. Workers who answer accurately can be accorded the qualification, enabling them to go on to perform tasks where the answers are not known.

This provides a rigorous method of assessing accuracy. However, it runs the risk that once workers have qualified, the quality of their work declines (because they know they have “won” the qualification already) and hence their accuracy declines, at just the point they start performing tasks where there is no objective yardstick to measure the output against.

An alternative variation is to qualify workers whose answers commonly agree with those of other workers. The danger here, however, is that workers become qualified based on how frequently they give the “average” answer, rather than necessarily the “right” answer. If most of the workers are giving an incorrect answers, future workers will be judged on whether they agree with those inaccurate workers, leading to an accuracy “death-spiral”.

3. Mix a set of tasks with known answers in with the unknown answers

Another technique (to be used instead of or alongside pre-task qualifications) is to mix a set of known tasks into a batch of mostly unknown tasks. An example here would be to make 5-10% of the tasks be questions where the correct answer is already known. This circumvents the problems identified above, because:

1. The accuracy of workers completing the tasks is measured against the “right” answers, not against the “average” answers from the pool of workers, preventing the accuracy “death-spiral”, and:
2. The workers cannot distinguish between those tasks which we use for qualifying and those that we do not, incentivizing them to continue completing all of the tasks accurately

Measuring accuracy

Let us return to the example that we’ve been working on at Keplar: namely checking a dataset of short content items to ensure that each item is in the language that we believe it to be in, so we can use the dataset to train a language detection bot. As you might remember from our previous post, each Mechanical Turk HIT should be “as small as possible”, and we should ask workers to answer closed rather than open questions. So rather than ask “what language is this content in?”, we ask e.g. “is this content in French?”  The purpose is to end up with a data set of thousands of content items that we are very sure are French.

For each HIT on Mechanical Turk, then, there are four possibilities:

1. Content is in French, and worker confirms it is in French – let this possibility be “FY”
2. Content is in French, and worker identifies it as not French – let this possibility be “FN”
3. Content is not in French, and worker identifies it as in French – let this possibility be “GY”
4. Content is not in French, and worker identifies it as not in French – let this possibility be “GN”)

The 4 possibilities can be mapped on a tree diagram:

We created a batch of tasks that asked workers to confirm whether or not a content item was in French. We specified that five workers should examine each item.

In our example, our primary concern was that an item not in French is incorrectly classified as being in French, because the workers fail to notice that it is actually in, say, German. Hence, the possibility that worries us is labelled “GY” on the diagram. We were much less concerned if a content item that is really in French is incorrectly classified as “not in French” (“FN” on the diagram), because disregarding that result simply reduces the size of our output dataset, but not the quality or reliability of it. So our worst-case scenario was that the 5 workers who answer the HIT “is this content item is in French” all got it wrong and said “yes” (“GY”) when the answer was really “no” (“GN”).

Because we were interested in false positives (“GY”s), we measured the likelihood of a worker incorrectly identifying a non-French content item as French. Hence, the 10% of known tasks included in our batch were all content items that were not in French (i.e. known “G”s).

For each worker “n”, we calculated the % of known, non-French content items that they correctly identified as not French (“p”) and the % of known, non-French content items that they incorrectly identified as being in French (“q”). To restate:

where is the outcome where worker n answers “No – this content item is not in French”,
and are the measured probabilities for worker n based on their responses to the known “G”s.

Results from workers who scored very low for accuracy (e.g. p<95%, q>5%) were rejected, and these tasks were listed again on Mechanical Turk, to be completed by new workers, whose accuracy was again measured. Eventually, each HIT had been answered five times, in each case by a worker who accurately identified at least 95% of the known bads in the data set as “not French”.

Across the entire data set, we calculated the % of known, non-French content items that were correctly identified as not French (“P”) and the % of known, non-French content items that they correctly identified as being in French (“Q”). Because we have rejected all workers where p<95% and q>5%, we know that P>95% and Q<5%. To restate this mathematically:

where and are the measured probabilities across all workers, based on their collective responses to the known “G”s.

Based on this figure, we can work out the probability that if a content item is not in French, all 5 workers would incorrectly classify it as French:

Then, assuming Q = 5% (in actual fact, it must be equal or less):

That means that there is a less than 0.0001% chance that if a content item is not in French, all 5 workers confirm (erroneously) that it is in French.

More generally, if the probability (as measured on a set of known tasks) of inaccuracy on a task = Q, and n workers are asked to confirm the result, and all of them agree, the probability that all the workers are wrong is given by:

Returning to our practical problem: we want to know how certain we can be, given that “n” workers have each independently confirmed that a particular content item is in French, that the content item is in French (i.e. that they are not all wrong). So we want to know the probability that the given update is in French, given that “n” workers have independently verified that it is – in other words:

Note that the above two equalities follow from conditional probability. The resulting function is of the form:

where
and

is incredibly small: it is the product of the probability that a content item is not in French (which should be less than 0.5) and . As a result, whatever the value of , the probability of a content item being in French when all the workers indicate that it is, is going to be very close to 1. For example, if:

and (which would be surprisingly high)
and (which would be surprisingly low),
then

In summary

By mixing a set of questions with known answers into a batch of Mechanical Turk questions with unknown answers, companies can track the accuracy of workers and take a statistical approach to measuring their confidence level in the results produced by those workers on Mechanical Turk. This makes Mechanical Turk an extremely powerful tool for having sometimes-unreliable humans input into highly scalable – and reliable – business processes.

In the next post in this series, we will look at how to use Python scripting to start to integrate Mechanical Turk into your business processes in an automated, scalable way.

Chris Cox's presentation at F8 is essential viewing for anyone in product management

The purpose of this post is not to summarise the developments or hypothesise on the implications: there are plenty of pundits doing that already. In this post we explore the importance of product vision to successful product development, and use Facebook’s Timeline as an exemplar of best practice.

What is product vision?

Every digital product should have a purpose: this answers the question “why would someone use the product”. Whereas the product purpose is the reason someone would use the product today, the product vision determines why someone would use the product at some period in the future. (Typically in digital up to five years in the future: few people are brave enough to make predictions about how the digital world will look more than five years from now.) The product vision is a long-term aspiration which guides product development decisions in the short-term.

Product vision isn’t talked about much in digital product development. That’s mostly because developing a suitable vision is particularly hard in such a fast-moving environment. Indeed, commitment to a vision can be a hindrance, preventing companies from adjusting rapidly to markets because of commitment to an unsuitable roadmap and vision. Much of our work at Keplar involves helping clients to adopt an iterative approach to product development that emphasizes learning from consumers and, as far as possible, co-creating digital products with consumers, so that the products develop around the actual needs and behaviours of real users. This more “bottom up” approach can sit uncomfortably with the more “top down”, vision-led approach.

The two approaches are not incompatible, however. The vision needs to be high-level enough that it gives product managers space to make short-term design and development decisions around user behaviours, but still move the product (and the userbase) in the direction of the vision.  Also the vision need not be static: it can evolve as you learn more about your userbase. At its core should be an aspiration that is recognisable to users, just as it needs to be recognisable to everyone involved in developing the product on both the design and development sides.

Characteristics of a good product vision

1. It is an aspiration: the vision needs to be a stretch – if it describes a product that already exists today, then it won’t prove any help in driving the product to new heights
2. It is inspiring: if people are going to overcome the myriad challenges associated with delivering a stretch target, then they have to really believe in the vision. The world should be a better place when the vision is realised
3. It is about the consumer, not the product or the company developing the product: it is easy to state the purpose of a product in terms of commercial benefit (it is nearly always more profit.) However that’s not something that users are going to buy into, and it’s a terrible guide to making short-term product development decisions because it is often hard to link these to long-term commercial gain. It is much harder to build a digital product around a commercial vision than build a business model around a product vision. That is why the product vision needs to be about the consumer, and not the company developing the product

The case for Facebook introducing Timeline: a complete overhaul of the profile page

Timeline is a significant evolution of the Facebook profile page, a core part of the Facebook experience since its inception.

Before we look at timeline in any kind of detail, however, it’s worth stepping back and examining the case for Facebook to significantly overhaul its profile page. The business case for doing so is very, very weak:

1. Facebook is the world’s largest social network, with a userbase of over 800M users. It is still growing rapidly, and it is hard to imagine that changing the profile page will drive significant incremental growth beyond its current growth rate
2. Facebook is only just starting to exploit the monetization opportunities around its existing services. It has developed into one of the biggest forces in display advertising (because of the large number of impressions and high eCPMs generated by banners on its site). At the same time it is developing an enormous database of its users and their relationship to a whole range of products and services, through the Like button; the possibilities to monetise this data are significant. So, there are plenty of short-term commercial opportunities that Facebook could concentrate on exploiting: it is not clear that overhauling the profile page would help it do so any faster
3. Overhauling the profile page is fraught with risk. Because it is a core part of the Facebook experience, any changes to it which jar with users are likely to lead to a massive outcry across the 800M userbase. Any change is going to take time for users to get used to, and in the interim their engagement levels may well decline. Fear of upsetting existing users is one reason why eBay’s user interface looks almost the same today as it did 5 years ago

Given the lack of commercial necessity and high risk associated with overhauling the profile page, few companies in Facebook’s position would take this step. But Facebook is no ordinary company.

The vision for Timeline

The vision for Timeline can be summarised:

The vision for Timeline meets all of the criteria for a great product vision. It is a huge stretch: whilst the current Facebook profile is probably one of the nicest profile pages on the web today, it is far from the “best place to define who I am”. The level of ambition drives each of the subgoals for the vision: the Timeline should encompass a user’s entire life, whilst also looking beautiful, being easy to populate and giving users the control they require. The vision is inspiring and if executed well will prove enormously exciting to Facebook users.

The impact of that vision

The resulting product is pretty spectacular:

And the competitive landscape has been transformed. Google Plus, which so recently looked like it was making significant headway in catching up with Facebook, now looks antiquated by comparison. Product vision may be all about the consumer, but the commercial impact of getting it right cannot be underestimated.

1. Conceptually to think about using Mechanical Turk
2. Use the web UI Amazon provide to do the actual implementation

Define your HIT(s)

At the heart of every Mechanical Turk engagement is what Amazon calls a “Human Intelligence Task” or “HIT”. Each HIT is an independent unit of work.

As we mentioned in our last blog post, we have been using Mechanical Turk to check the language of a short content item. We already have an inkling what language each content item is, however we are only 70-80% sure that we are correct – so we use Mechanical Turk to get real people to check if each guess is correct.

In our case, then, each “HIT” consists simply of a worker checking the content and either confirming that the content is in the language we thought it was, or not. Notice this HIT has several important characteristics:

1. It is a very simple to instruct. “Look at the below sentence. Is it in French? If so, click ‘yes’. Otherwise, click ‘no’.”
2. It is independent. We have millions of sentences to check. However, the checking of each individual sentence is a completely independent task: there is no requirement that the person checking sentence A needs also to check sentence B. Hence it is possible that many hundreds or thousands of workers can work on the tasks in parallel, to ensure they are done quickly.
3. It is repeatable. We can ask a number of different workers to perform the same task, and they should all give the same answer.  (This becomes important for ensuring the accuracy of results, because it means that we can verify the accuracy of individual tasks and individual workers.)

Best practices for defining HITs

1. Make your HIT “as small as possible”

Because you pay workers on a “per hit” basis, it is tempting to get workers to work harder to complete each HIT, for example by asking them to provide more information to complete each HIT. (For example, asking them not just whether or not a webpage contains pornographic content, but also whether it also contains expletives.)  Avoid this temptation! It makes the instructions more difficult to follow (which increases the probability of inaccurate results), makes it harder to assess accuracy (because you have to assess the accuracy of two answers per HIT rather than one) and does not save money (because workers expect more money for HITs that take longer).

Instead, ask just one question per HIT. If there are two questions you have of a specific content item / web page / anything else, create 2 HITs. This keeps each HIT simple and allows you to adopt different strategies to ensure the accuracy of answers for each question.

2. Ask closed questions rather than open questions

Instead of asking “what language this sentence is”, ask “is this sentence in French?”.  Closed questions are preferable for a number of reasons:

• The results are easier to analyse. HITs are performed by humans, but the output should be machine-processable if you plan to use Mechanical Turk as part of a set of scalable business processes.  That will be much easier if only one of a finite number of results are possible for each HIT
• It is easier to assess the accuracy of the results
• Closed questions are (normally) easier for the workers to answer. This makes it more likely that they will give accurate answers

Publishing HITs on Mechanical Turk

Amazon provides 3 ways to create HITs, publish them and collect the results: a web UI, a command-line interface and an API. When getting started, the web UI is the simplest interface to use.

1. Set the basic parameters of the HIT

Before creating a new HIT template, Amazon makes users fill in a long webform. Some guidance on the individual sections:

Describe your HIT

HIT “Title”, “Description” and “Keywords” should all be filled in to make it as easy as possible for workers to find your HIT (by searching for appropriate keywords e.g. “French, language”) and define the task to do as clearly and unambiguously as possible.

Working on your HIT

Overestimate the amount of time allotted to each HIT. Workers are typically good at looking at a couple of sample HITs and making a guess as to how long each will take – they will use this figure (rather than the figure you give) to calculate the effective hourly wage you’re paying. So the only real impact of filling in this field is to cut workers short when they’re doing a task: for that reason it’s always best to give a much high number than is realistic.

Amazon also offers a set of options for ensuring that only workers with a specific history of accuracy are allowed to fill in your results. There are a number of different strategies to ensuring accuracy, we will discuss these in a later blog post. To start, we suggest not limiting HITs to “Masters”. There are other, more reliable means, of ensuring accuracy.

Paying workers

There are no hard and fast rules on what you should pay workers. The more you pay, the more likely workers will be to prioritise your HITs and the faster they will be completed. We generally look to pay workers at least $5 per hour, and have been impressed by the speed with which work has been completed at that rate.

2. Create the HIT template

The second step is to design the “HIT template”: the webform with the question each worker will answer. Assuming each HIT involves asking a closed question, Amazon provides a visual designer to make creating the form straightforward:

The corresponding HTML for the form is super simple, and shown below:

<h3><span style="font-family: Arial; ">Is this Twitter update in French?</span></h3>
<pre><span style="font-family: Verdana; ">${Message}</span></pre>
<table cellspacing="4" cellpadding="0" border="0">
  <tbody>
    <tr>
      <td valign="center"><span style="font-family: Arial; "><input type="radio" name="is_language" id="is_language_yes" value="yes" /><span class="answertext">Yes this update is in French</span></span></td>
    </tr>
    <tr>
      <td valign="center"><span style="font-family: Arial; "><input type="radio" name="is_language" id="is_language_no" value="no" /><span class="answertext">No this update is not in French</span></span></td>
    </tr>
  </tbody>
</table>

Note the ${Message} placeholder. Before we run the HIT, we will upload a CSV file into Mechanical Turk. Each line in the CSV will represent one HIT, and each line will therefore need to contain a cell with the actual message who’s language we want to confirm. The title of that column will be “Message”, and Amazon will automatically insert that content into the webform where the ${Message} placeholder is found.

3. Upload the relevant data to Amazon and publishing the HIT

Once the template has been created, you need to upload the data to serve into each webform.

At the very least, the CSV you upload should have one column for every placeholder in the webform – in our example above, there is only one, ${message}.  It is also possible to introduce additional columns of data: for example, if the data comes from more than once source, you may want to indicate which source in a “source” column in the CSV. The source column will not impact the HITs. However, when the data comes back from Amazon (with the HIT results), the source fields will be included, and this will make it easy e.g. to analyse the accuracy of different sources.

Note: Amazon is very fussy that the uploaded file is UTF-8 encoded. For that reason, we recommend not using MS Excel to generate or edit CSV files used with Mechanical Turk, because Excel has a tendency to mess up the character encoding.

Once the data is uploaded, you can preview the HIT, as below:

Assuming you are satisfied with the HIT as previewed, you are in a position to go live!

4. Downloading the results

Once workers have completed all your HITs, you will want to download the results. Again, Amazon makes this straightforward via their Web UI:

5. Next steps

Using the results: approaches to accuracy

Clearly there’s no point in going to the effort (and expense) of using Mechanical Turk unless you use the results to do something useful. Quite what that is depends on your use case – but in most use cases, you’ll want some way to ensure that the results are accurate. This is a big topic: we’ll discuss one particular approach to accuracy in a forthcoming blog post.

Completing the Mechanical Turk workflow: approving HITs and paying workers

Before workers are paid, you have to approve the work that they’ve done. That approval process should be part of your general approach to accuracy – again, we’ll be covering this in a subsequent blog post.

Assuming, though, that you have verified the accuracy of the work, you will need to approve each HIT. In the downloaded CSV, Amazon will have included an “approved” column. The simplest way to approve HITs is by marking an “x” in this column for each approved HIT, and then uploading it back into Mechanical Turk. Amazon will then deduct the funds from your account, and pay the workers accordingly.

Stay tuned…

Hopefully this blog post has given you the encouragement to start experimenting with Mechanical Turk to automate some of your own business processes using HITs. Feel free to share your experiences getting started with Mechanical Turk in the comments below, and in the next post in this series we will be looking at strategies for improving the quality of your results.

Amazon’s Mechanical Turk has been around for some time (it was first launched in 2005). However, we’re surprised by the number of businesses we encounter who are still not aware of Mechanical Turk and the potential ways using it could save them time and money, whilst opening up new product development opportunities.  In this blog post series, we look at what Mechanical Turk is, how it works, where it works best, how to start using it and how to build scalable systems around it that effectively yield accurate results.

So what is Mechanical Turk?

Mechanical Turk is two things:

1. A huge market of remote workers eager to do well-defined tasks
2. A set of tools for engaging with that worker community, making it easy to (i) set them tasks, (ii) collect the results, (iii) assess the accuracy of those results, (iv) pay them for their time

Mechanical Turk is perfect for situations where a company has a big job that needs doing and:

1. The big job can be broken down into a large number of independent steps
2. Those independent steps are better performed by a human than a computer. Typically this means applying a value-based judgement for instance: “is this suitable for work?” (content moderation), “is this in Italian?” (language detection), “is this current affairs?” (categorisation)
3. Those steps are all of a particular type (or a set number of types).  In other words, a single set of instructions is suitable for a large volume of tasks / work

Amazon gives several examples of potential tasks, including categorising items and or moderating content. Categorising items is a famously difficult thing for machines to do (often requiring sophisticated semantic analysis), but it is something that people do very naturally. (As Wittgenstein noted in the case of working out what does and does not constitute e.g. a “game“.)

At Keplar, we have been using Mechanical Turk to check the language of specific items of content in a large data set, which itself is being used to train a semantic engine. Whilst it is possible to use computer programmes to ascertain the language of each content item (Google Translate, for example), these systems are not accurate enough for our purposes: only humans can provide the accuracy we require, and Mechanical Turk is one of the few places we can easily get the volume of people, with the requisite language skills, to quickly and efficiently check the large number of data points necessary to build a clean data set.

In the remaining three blog posts in this series, we will explore:

1. How to get started with Mechanical Turk
2. Approaches to ensuring accuracy
3. Best practices, tools and technologies to build scalable systems that leverage Mechanical Turk

The minimum viable product is that version of a new product which allows a team to collect the maximum amount of validated learning about customers with the least effort

Lean startup and MVP theory started as a response to the “big bang”, capital-intensive approach taken by many businesses (especially VC-backed startups and public companies) to developing new digital products. The observation underpinning MVP is that pre-guessing what customers will want and building a big bang product to meet all of those expectations is risky – because only by interacting with those customers can you really understand their needs and thus what your product should do. As the Prussian field marshal von Moltke put it first:

No plan survives contact with the enemy

The danger of the big bang approach, then, is that you discover your customers’ needs far too late: instead of steadily iterating a live minimum viable product towards optimal product-market fit, you are left needing to drastically re-build your product – often at great cost – to finally start meeting those customer needs that you just discovered.

At Keplar we are strong proponents of the data-driven, iterative approach designed around an initial minimum viable product. It’s certainly a methodology with strong opinions – and not always comfortable ones for clients grown used to the big bang approach promoted by full-service agencies and IT consultancies. But the fact is that those big bang projects often fail, and as a methodology which seeks to reduce capital expenditure and maximise the product’s customer appeal, the iterative MVP-based approach is an increasingly compelling one.

Where, however, we see a lot of room for confusion is in what actually constitutes “minimally viable”. Part of the problem here is that much of the thinking about MVP has come from lean startup practitioners who have a few traits in common:

1. They are often launching products into “blue ocean“ market spaces where no real competitors or proxies exist
2. They often have no prior expertise in the relevant sector (e.g. fashion, education) or function (e.g. analytics, online retailing)
3. As technologists they rarely have backgrounds in customer-facing roles
4. They are often planning to grow using fixed-cost marketing channels (word-of-mouth, viral features, SEO) rather than cost-per-user channels (SEM, display advertising, affiliates)

Taking all of these factors together, you can see why the minimum viable product for many startups is quite, well, minimal: there’s typically very little that the founders know (or can know) about the customer needs – and thus the product – before they launch. Furthermore, with a “blue ocean” product launch a startup isn’t too worried about disappointing customers with too little functionality – because some functionality is better than none. And finally, even if the initial launch product does prove a turn-off for some customers, the use of “lean” fixed-cost marketing channels means that there is no direct financial loss incurred from failed customer conversions or user churn.

For established, experienced businesses not launching into blue ocean markets, we believe that the minimum viable product is actually much closer to 70-80% of the hypothetical end product than it is to a lean startup’s 30-60% mark. This is because businesses such as these tend to diverge from lean startups on all four of the traits listed above:

1. Established businesses are rarely undertaking all-new, blue ocean product launches – instead, brand and product extensions, ”fast follower” copycat launches and new business models for existing products are the order of the day. In all of these scenarios, the company is competing with other players (and sometimes even with themselves) – so the launch product has got to pass the customer’s crucial “smell test”: is this product a credible alternative to the others on the market that I could use?  And if it doesn’t pass that test, then the company can’t start collecting the behavioural data from customers that it needs to iterate the product further
2. Established businesses typically have significant sectoral and/or functional expertise – and they can leverage this domain knowledge about their sector or function to significantly reduce the number of unknowns that require testing with the MVP. To take the example of an accounting software provider looking to launch a new hosted SaaS product: they already know exactly how the core product should work, so where they will need to iterate post-launch will be largely around pricing and the potential new SaaS-only functionalities (e.g. hosted backup, online collaboration)
3. Established businesses already have a good understanding of their customers’ needs – of course an established business has far less to discover about their customers’ met and unmet needs than a new market entrant; moreover their existing customer relationships often enable them to test customer attitudes to a potential new product prior even to designing the MVP
4. Established businesses will often put significant marketing spend behind the new product – to establish the new product in the market and acquire users, companies will often commit to a significant marketing spend on SEM, display advertising, affiliate marketing and similar. However if the MVP is too minimal and doesn’t pass the customer’s “smell test”, then that marketing spend is effectively wasted – and in fact might even translate into negative word-of-mouth for the company’s brand

Putting all of these reasons together, it is clear why for many companies the minimum viable product will actually be a semi-complete product requiring a considerable development effort. This MVP should not however be confused with the big bang approach, because there is always a gap in customer understanding which requires data-driven investigation and product iteration post-launch. And in fact the increased capital investment in bringing a 70-80% functional MVP to market makes the post-launch approach to iteration and optimisation even more important.

To summarise, at Keplar we believe that for many businesses operating in (or moving into) established markets, it makes more sense to design, build and launch a “Minimum Sticky Product”. We would define this “MSP” as follows:

The minimum sticky product is that version of a new product which allows a team to gain traction in a “red ocean” market and close the gap in customer understanding with the least effort

A minimum sticky product, then, is one which is just good enough that customers stick with it in the face of competent competitors – in other words a product that starts to get traction in the market, which collects that crucial behavioural data from customers and which can justify per-user acquisition marketing thanks to low initial churn. If you’re already building a minimum sticky product like this, let us know in the comments below!

If you are developing a new digital business or product and would like to talk further about designing, building and launching your “Minimum Sticky Product”, don’t hesitate to get in touch.

What is a mock-up?

A mock-up, simply put, is a sketch of how individual screens on the digital product (be it an desktop application, mobile application, web app etc.) might look.  It is a rough sketch:  the point is not to indicate how the site will look graphically, but how it will work functionally.  It should contain all the buttons, menus, inputs and other functionality that each screen will have.

Here is an example mockup, one of a series created for a B2B client of ours:

A single mock-up is not especially useful, but a collection of the different screens that make up an application are enormously valuable.  That’s because collectively, they can be used to see how a user would step through the different workflows that make up an application.  (Indeed some mock-up tools let users connect individual mock-up screens to create interactive prototypes of finished products.)

The value of mock-ups

1. Turning customer and commercial requirements into a digital product: fast

There’s a creative leap that product managers need to make to turn a set of commercial and customer requirements into an actual digital product.

Mock-ups are an excellent tool for making that leap.  Because different screens can be quickly sketched out, they make it easy for the product manager to start to visualise how the product might work.  Better, because they are so quick to produce, the product manager should be able to put together multiple different versions and explore the pros and cons of each.

2. Building consensus around a digital product design from different stakeholders

The most powerful feature of mock-ups is that they are easy to understand.  That means that the product manager doesn’t have to make the creative leap by herself: she can share it with other stakeholders including the management team, marketing and business development teams, engineering team and customers, get their feedback, and even have them contribute their own mock-ups to the process.  This increases the scope for creativity, and ensures that the finalised design meets all the different stakeholder requirements.

3. Getting users / customers to input into product development as early as possible

As well as enabling product managers to include internal stakeholders in early product development design decisions, mock-ups (and especially prototypes) enable product managers to start testing designs with customers themselves.  This take a huge amount of risk out of the development process that the product, on delivery, will effectively meet customer needs, whilst giving customers a chance to contribute creatively to the product design process and influence fundamental design decisions.

4. Starting the product iteration process before the development process

We discussed in the last blog post how the best digital products are constantly iterated, on the bacl of web analytics and customer feedback.  Because customers can be involved earlier using mock-ups, initial iterations can be made based on customer feedback.

Because this happens before the development process has, this gives the product manager the maximum flexibility to take customer feedback on board before committing any development budget.

5. Building a technical architecture that meets workflow needs and is flexible enough to adapt

Mock-ups allow the product to progress through several iterations before the technical team must settle on a technical architecture; this means that the technical team will be better placed to select an architecture that, rather than just reflecting current needs, is flexible enough to evolve in the likely directions the product will develop.

When in the product development cycle digital products should be used?

As early as possible!  Generally, product managers should start to mock-up potential designs as early in the development process as possible – potentially even before budget has been committed to a product, to help give stakeholders an idea of whether an idea is worth investing in.  Mock-ups are best used alongside user stories, so that different stakeholders understand the motivations for different people using the product, and how the workflow in the product meets those user needs.

How should mock-ups be produced?

There is a wide range of applications that let people quickly put together mock-ups, and indeed, some people use good old fashioned paper and pencil.  At Keplar, we use Balsamiq Mockups, because it is super quick, super light, and super simple.

Need help turning your commercial and customer requirements into digital products?

At Keplar, we help clients develop and execute digital strategies, including designing digital products (using mock-ups).  For more information, drop us an email.

In the previous post on product management, we took a high level look at the role that product managers play and the value that they provide. In this post, we go into a bit more detail, but from the point of view of companies that outsource the building of digital products (including websites, iPhone apps etc.) to digital agencies. In many (although not all) of these cases, there is no designated product manager (at either the client or the agency) and as a result, the effectiveness, popularity and commercial success of the digital product will suffer. By looking into the kind these issues in more detail, we hope to explain in more detail what it is that makes product management so critical, and also to provide an approach for companies engaging with digital agencies to avoid these pitfalls.

A typical scenario

A company that makes the vast majority of its money offline wants to do something online.  Maybe they’re a retailer looking to start selling online, or a magazine publisher wanting to make their content available online.  The client, unfamiliar with online, approaches a digital agency to help them.  Because the client is unfamiliar with online technology, they look for a digital agency that has done similar websites for similar clients.  The digital agency sits down with the client and run a series of requirements gathering workshops to understand what the client wants.  They put together long requirements specification and work with the client on a graphic treatment for the site.  On the basis of the images of these visuals, and the requirements specification, the agency builds the client its website.

It’s worth stressing that no one at the client is a designated “product manager”.  Someone will be designated as project manager, who has to work with the agency to ensure that the site is delivered to specification and on time.  But that person will not have the responsibilities associated with a product manager that we listed in our last post in the series:

1. Providing a high level vision for the product
2. Working with user interface experts to “fill in” the product vision
3. Providing a roadmap for the product’s development
4. Monitoring and adapting the product roadmap based on new insight
5. Provide a “customer-minded resource” for the rest of the company, especially the developers building the product, to draw on

Similarly, no one at the agency will be a designated product manager.  With no access to a product’s potential users or the community managers who have user relationships, or a client’s web analytics, no one in the agency is in a position to represent “users”.  Similarly, they are not best placed to represent the interests of the different stakeholders at the client  (management, marketing, BD, sales etc).

What are the pitfalls of proceeding in this way, without a product manager?

Pitfall 1:  the digital product doesn’t work as well as it could

To ensure a digital product works well, a product manager works closely both with people in the business to understand the commercial objectives for the digital product, and prospective users for the product to see how it works for them.  This knowledge is key to the product design.  Further, this is an ongoing process:  upfront work is required before the product is specified, but prototypes and early product builds should be tested with real users to inform design decisions.

On the client side, no one is assigned responsibility for this, and no one is trained to do it, so no one does it.  Clients typically are not great at specifying digital products because this may be the first digital product they’ve built.  They know they want an online shop and they expect it to work like other online shops they’ve used.  (So have a shopping basket, catalogue, checkout process etc.)  But no one at the client is likely to have stepped back and thought:  what is it that makes the buying process special for our customers in our sector? What are the key criteria customers use to make buying decisions in our sector, and what can we do to encourage them to buy from our site? Are these people who know what they want and need to find it and purchase it as quickly as possible?  Or do we need more of a browsing / window shopping experience?  How often do customers make repeat purchases, and what is the relationship between one purchase and the next?  How long is the buying cycle, and what are the key points in the buying cycle we need to facilitate that will help our users and drive our bottom line? What kind of relationships do our customers have with us, and how should the workflows be designed to reinforce those positive aspects of our relationship with them?

On the agency side, the team will be preoccupied with delivering the project on time and on budget.  The closer the digital product looks and works to “similar” projects that they’ve done before, the easier the project will be.  The agency also know that the client will inevitably compare their site to those similar pieces of work that drew them to the agency initially.  So why would an agency force a client to take a step back and challenge the client’s assumptions around commercial dynamics and user behaviours before embarking on the build?  And if it looks like the client is happy with what the agency is building, why would an agency encourage them to test it with users and potentially find out that it won’t be effective?  Better to let them find out later (once the project has been signed off and they’ve been paid), and win follow-on work off the back of that insight.

Typically, the agency will be judged by the look of the completed project.  For that reason, they will be strongly incentivized to put a lot of work into the graphic design of individual pages.  (And digital agencies, in the UK certainly, are dominated by graphic designers.)  But the agency is not incentivised to do all of the underlying ground work on which a good graphic design depends:

Taken from "Lean Product Management" by Dan Olsen. Dan took the framework from Jesse James Garrett’s “Elements of User Experience” chart, free at www.jjg.net

Because no one is being judged on conceptual design, information architecture or interaction design, no one makes sure that these are done right. The product produced would inevitably be much better designed, and hence more commercially successful, if someone did.

Pitfall 2: scope creep, project delay and budget overrun

Part of good product management is about product design.  But part of it is also about process:  managing the process and stakeholders around the development of digital products to ensure that the needs of the business and customers are met, and any conflicts / trade-offs between different stakeholders are managed.

It is highly likely that when the digital product is developed, people in the business will review it and want changes to be made.  The agency will point at the specification and show the client that this requirement was never specified, hence it can be delivered, but that will cost extra and will require extra time.  The client will be upset, because they’ll feel their request is reasonable, and it wasn’t obvious to them when they read the requirement spec that that was how the requirement would be met in practice.  So both the client and the agency suffer.  With no one single person who has a holistic view of the value to the business of an individual feature, and the engineering cost associated with implementing it, no one is in a good position to manage that trade off.  (The best product managers at digital companies manage the tension between commercial folk and engineering staff, and use that tension creatively to drive exceptional product design.)

Avoiding delivery delays and budget creep is typically the responsibility of the project manager, and often there will be one at both the client and at the agency who’ll be in trouble.  But in this case it’s not their fault:  it’s the lack of a product manager that’s to blame.  A good product manager will have flushed client requirements out as early as possible, and be commercially and technically literate enough to handle the client-digital agency tension that arises when previously unknown requirements do inevitably come to light.

Pitfall 3: product stagnation

Finally the digital product is delivered:  it’s not as good as it should be, it cost more than it ought to, and it took longer to deliver than it should.  Now what?

Launching the product is a huge opportunity to test the assumptions that went into its design, understand what works and doesn’t work and start to map out a longer term development process to drive more business value out of it.  Instead, it will typically stagnate.  No one is responsible for managing the product’s development at the client.  Maybe no one is trained in using tools like web analytics and customer feedback surveys to understand where it does and doesn’t work, and how to turn that insight into future product design features.  By this point the agency has washed their hands of the project.  They’ll be back – but normally when they have a shiny new competitor website to showoff, rather than a new insight about the client’s own business and how online is supporting their commercial strategy.

At the very moment where the client could be really learning about their customer’s behaviour, and using that knowledge to drive growth in their digital venture, the process stops.  The budget is spent, the digital agency are gone and everyone wants to move on.

Avoiding the pitfalls

Fortunately, all the above pitfalls can be avoided, to the benefit of both the business and the digital agency.

The best step a company building a digital product can take is to hire an internal product manager.  An internal person is in the best possible position to do a good job of the product management because:

1. They understand the business
2. They have ongoing relationships with the different internal stakeholders / departments
3. They understand the customer base

Sometimes that won’t be possible.  This is especially true for companies that are dipping their toes in the digital space, but who are not fully committed, or have insufficient financial resources for a full-time product manager.  In this case, it is possible to outsource the product manager role to a company like Keplar.  To do this successfully:

1. Do not outsource the role to someone at the digital agency.  The product manager role critically involves managing the tension between commercial requirements, customer / user behaviours (what the people who use the digital product actually do when they use it) and the development team.  Most digital agencies do not have the commercial or analytics expertise to represent either user behaviours or commercial requirements.  (A good agency is instead expert at rapidly building extendable digital systems, leveraging the right technology platforms and using the right processes to ensure that everything built is built quickly and is thoroughly crash tested.)  In the odd occasion where they do have the capabilities internally, the very fact that they work for the development agency means they will inevitably manage those tensions in the interests of the digital agency, rather than the client.
2. Ensure that there are designated internal staff who after the product has been built, have the necessary skills and resources to assess how well it is working, and spot where there are opportunities to improve it.  The outsourced product manager should train internal staff to use web analytics and customer feedback on an ongoing basis.  That means that even if there is no internal resource that can use that understanding to adapt the product roadmap, the company is still in a position to engage with the outsourced product manager, and the digital agency, again at a later stage to incorporate that insight.
3. Minimise budget on an initial product build, with a view to iterating it.  This is counter to the approach of many digital agencies, who will work with the client to build the most impressive v1 possible.  Unfortunately an impressive v1 tends to be expensive, and leaves the client with little appetite to adapt and iterate the product, even in the face of all the new insights (e.g. around customer behaviour) which the v1 yields
4. Utilise visual prototyping tools, with internal staff and real users (where possible) to drive requirements gathering and product design before building the v1. Visual prototyping is an incredibly powerful technique for flushing out internal requirements and beginning to get a handle on how real customers behave, before any development budget is spent.  We will cover visual prototyping tools in more detail in our next post in this series.

If you are looking at building a digital product for the first time, or developing one you already have drop us an email to explore further how putting product-management at the heart of your new product development could drive your bottom line.

Here at Keplar some of our recent advisory work has been for online community sites, helping them to improve their customer proposition while also growing their revenues. In the course of these projects we have started to develop a new technique for guiding the development of these sites which we call “value chain mapping” – a technique that works to improve the way community sites serve their community whilst simultaneously growing their revenue. This blog post aims to explain this technique – articulating the theory but also providing practical advice for owners of social networks. (None of the companies mentioned in this post are current or past clients of Keplar.)

Vertical social networks: a primer

For this post we focus on “vertical” social networks – these are simply community-oriented websites which focus on one content area, interest group or demographic. We use the word vertical to distinguish these community sites from general-purpose social networks such as Facebook and Bebo. For this post we use Mumsnet, the UK website “by parents for parents”, as our main example.

So, what does a vertical social network look like? Typically these sites provide their user community with a set of “soft tools” such as forums, profiles and articles, all designed to support the site’s users in socialising, sharing knowledge and fulfilling specific needs. On Mumsnet, those user needs would include childminding, keeping their child healthy, and finding a kindergarten.

On the monetisation side, vertical social networks generate most of their revenues through advertising – showing users targeted or general banner ads but also directing users towards relevant businesses for a fee (see for example the Mumsnet Mall). It’s a delicate balancing act for the site owners as communities respond badly to advertising which they deem excessive, irrelevant or dishonest.

So, what’s wrong with this model?

The current model is far too narrow

The fundamental weakness of this approach to a vertical social network is that it’s far too narrow. To continue with the Mumsnet example: parents are an important part of the parenting ecosystem, but they are only one part of it. There are many other people and organisations who have a direct or indirect role to play – from the children themselves through to care providers (e.g. childminders, paediatricians), merchants (e.g. toy shops, pharmacists), manufacturers (e.g. buggy makers) and educationalists.

In short, parenting is a huge, complex ecosystem of which parents are the most important part – but they are only one part. And a site which only gives a voice to one player in an ecosystem is an echo chamber – think of the House of Commons without the Opposition, or eBay without the sellers. These sites are facilitating peer-to-peer communication between their narrowly defined interest group, but they don’t truly fulfil that group’s needs, because they don’t allow them to build real relationships with the wider ecosystem.

Our approach for vertical social networks, then, is built around the idea that these sites need to expand to accommodate – and serve – as much of their respective ecosystems as possible. It is only by this expansion that the user needs of their core interest group (e.g. parents) can be properly met.

Mapping value chains

If this is the theory, then how does a site such as Mumsnet go about incorporating its wider ecosystem – isn’t this a Herculean task? Fortunately, each ecosystem breaks down into a set of smaller, discreet value chains. By a value chain we mean a set of people or organisations which pass value in one direction to fulfil a user need, for example the value chain for childminding looks something like this:

Our approach is to incorporate these individual value chains into a vertical social network to better serve these user needs. We do this by “mapping” these value chains into the network – in the case of childminding this would be a three step process:

1. Extend the network to allow agencies and childcarers to have individual profiles on the site
2. Provide discovery tools to allow parents to find and evaluate these childcarer profiles on the site
3. Provide market tools to allow parents to buy childcare from providers through the site

With this value chain mapped into the social network, parents are now able to discover and buy childcare without leaving the site. We are leveraging all the powerful attributes of social networks – discovery, reputation, communication – and applying them to solving a specific user need. Incorporating more of the ecosystem is then simply a case of “breaking off” more value chains, mapping them onto entities within the social network and providing the necessary tools to connect them to users.

If this is the approach, what is the impact on the site’s bottom line? The impact can be considerable, because whenever a service is fulfilling key user needs there are significant revenue-generating opportunities. To go back to childcare: in the UK alone this is a £4.1 billion market (source). If a social network can model a value chain like childcare then there are various ways of capturing a slice of that value – perhaps a small slice, but certainly much higher than any revenues from current advertisers from that value chain. Wahanda, the health and wellbeing network, is a great example of a site which has mapped its key value chains and is capturing value as a result.

Conclusion

In summary, then, value chain mapping is a way for vertical social networks to break out of their current narrow focuses and start comprehensively fulfilling their user’s specific needs. In doing so they can directly capture value from their respective ecosystems, in addition to the indirect revenues they currently make from advertising.

A social network can start small with this approach – we recommend that network owners reading this start with just one user need: take that user need, understand the value chain that fulfils it and make sure that the relevant organisations and people have a space within your network. And let us know how it goes! We’d love to get a discussion going around this post with a view to building on this approach and helping social networks to deliver more.

Interested in monetising your community site better? Send us an email to find out how Keplar can help.

Dear Spotify,

We know that you are one of the most exciting companies of the moment.  Everyone, from the Guardian, to Techcrunch and e-consultancy is singing your praises.  In spite of all the hype, we actually believe that Spotify remains under-hyped.

It’s about more than your business model

Mainstream media interest in Spotify has focussed on your business model.  The music industry has of course been badly hit by the growth of the internet, the rise in piracy and the shift in consumer spend to video games.  As a result, almost all of the coverage has narrowly focused on this question of whether you can make the music industry profitable, and whether you can be profitable yourself.  People have, shockingly, compared Spotify to We7, shocking because We7’s sole purpose is to deliver ad supported music to consumers for free, and the Spotify proposition offers a whole lot more.

It’s about a whole new way of consuming music

It is tempting, given Spotify’s iTunes-like interface, to think of it as just another music delivery mechanism, one that’s preferable to downloads because it is easier to protect against piracy.  But that misses the point.  Streaming music allows you to potentially offer your users two new types of listening experiences, without parallel in competitor services.  One of these has already been launched, and I hope that we don’t have to wait long for the second:

1.  Transform my mobile phone into the ultimate MP3 player

Spotify’s premium service for iPhone, Android and Symbian is incredible.  I have access to (almost) any song, at any time, immediately.  It doesn’t matter that my iPhone has only 8Gb.  I have instant access to an enormous music catalogue, unparalleled by even the biggest private collections.

2. Social consumption of music

Spotify has barely scratched the surface of the social features they could build around their product.  Sharing playlists is just the beginning.  I want to be able to discuss tracks with other music enthusiasts as I listen to them (and we listen to them together, in real-time), lookup forthcoming gigs, browse other people’s libraries for new music, send tracks to friends, even potentially talk to the artist while we’re both online.

How does Spotify seize these opportunities?

Given the potential, it’s a little disappointing how quickly you are progressing with your music revolution. Some recommendations:

1. Start properly marketing / advertising the services you already offer

Your mobile phone app is nothing short of incredible.  It has serious mainstream appeal.  You should be all over mainstream media telling people: “we can turn your phone into the ultimate MP3 player”  Don’t get stuck, as Skype did, trying to monetise your cost conscious, tech-savvy early adopters, but rather start advertising directly to the thousands of people who would kill to have a limitless library of music in their pockets.

2. Develop social features – quick!

A flippant thing remark, perhaps, but it needs saying.  Those social features aren’t “nice to haves” – they’re key in differentiating the Spotify proposition from everything that’s gone before it.  And that’s important because music piracy means by and large people are much less willing to pay for just the vanilla experience of listening to individual tracks.  However they may well, for example, be willing to browse and listen to their favourite artist’s music collection – be that Rev Run or Paul McCartney.  Especially if they can do so with their friends, compare notes, and rate it.  And imagine if they could do it with the artist him/herself?

There are several challenges to developing the social features quickly.  One option you might want to consider is opening up your platform.  Your core audio streaming system is brilliant – why not follow the path of Facebook or Twitter and nurture an imaginative developer community to build a wealth of different apps to take advantage of that core technology?

And if it all sounds like a bit much?  Hire Keplar LLP

Open source is not a business model

Inspired by the ongoing battle between the European Union and Oracle over the future of MySQL, yesterday’s New York Times notes that “open source as a model for business is elusive”.  Sadly (and uncharacteristically), the New York Times is wrong.  Open source is not a model for business at all.

To start with what open source is: it is an approach to development that prioritises development speed and flexibility over intellectual property rights.  By giving developers around the world the ability to contribute and extend open source products, open source projects can cater to many more users and scenarios than might be possible.

Part of what makes a project open source involves giving the product away for free:  this encourages developers to take and extend products, safe in the knowledge that they do so legally.  That doesn’t make open source a business model, although it means that being open source has implications for your business model.

If you have to give the software away for free, then how do you make money from it? This is a question that many open source projects have not managed to answer satisfactorily.  The classic answer has generally been around providing value-added services, but sadly, service businesses don’t scale as well as typical (closed source) software businesses.  As a result, some of the most successful open source projects have found alternative revenue streams altogether:  Mozilla’s Firefox has benefited from funding by Google (by driving search engine revenue using the box in the top right half of the browser).  OpenX generates revenue by offering the publishers who use its free ad servers value added services such as a real-time ad exchange.

A more interesting question than whether or not open source is a viable business model is where does open source work best?

Open source-code is rarely the most interesting thing about an open source project

The most successful  open source projects are the ones that attract the largest, most talented developer communities to work on them.  And the projects that tend to attract the most developers are those that develop things that  developers want to use.

WordPress is a great example:  not only do many bloggers have some development skills which they can lend to the project, but equally the WordPress team have worked hard to build a product that can be easily extended by people with very limited development expertise. Firefox is another superb example – a huge number of developers have contributed plugins to make their lives (and the lives of other, non-technical users) easier.

In cases like WordPress and Firefox plugins, developers get to build tools for themselves and others.  All good application development should involve plenty of user feedback / product design iteration.  When the developer can use a tool, and then extend it herself, that feedback loop becomes quick to close, and iteration comes easily.  At that point, open source product development has a real advantage over its closed source cousin.

Where open source tends not work so well is when it is built for users other than developers.  Probably the best example is GIMP – an open source graphics package, popular with web developers, but barely touched by the graphic designer community, who prefer more user friendly applications like Photoshop.  It is not that people who are good at development are necessarily poor at UI (although they often are). It is that a developer on the GIMP would have to guess how a designer might like to use it – and that is if they even cared what designers wanted. Either way, that iterative feedback – product design loop becomes much more difficult to close.

So what now for open source?

Open source is here to stay.  That’s not because open source is a viable business model, but because making a product open source is a useful tool to drive adoption and application development, especially for companies that don’t have to rely on sales of that product for revenue.  (Google’s Android and Wave are two salient examples.)

Open source has also made a significant difference to the technology eco-system.  Companies with little or no open source DNA have seen the way that open source enables fast development and richer products, and they have emulated this with their own – proprietary – platforms. A great early example of this was how Salesforce.com opened up their proprietary hosted CRM solution to third-party developers. Now everybody from  Facebook and Apple to Google are following suit. We will be writing another post on how platforms have superceded the open source “business model” (although of course not open source itself) soon.

And if you’re running an open source project yourself? If your project is attracting useful contributors and fulfilling a user need that you either have yourself or you genuinely care about, then you have nothing to worry about. If on the other hand you are focused on a non-technical userbase and you have to pay developers to work your project, we would suggest you take another look at things. Are you really running an open source project – or is it a commercial software business that just happens to have a pricepoint of free and freely viewable source code? Whatever the answer, if you have a significant gap between your development costs and your service-based revenues, then you better keep being nice to your corporate sponsor or VC…