One of the Keplar bookshelves (the Scala books are out on loan)

Scala is a relatively young programming language which runs on the JVM (the Java virtual machine) and attempts to fuse Java’s object-oriented approach with a more Haskell-ish functional programming style. A few things made Scala stand out as the language for our agile data analytics work:

1. Scala runs on the JVM, which gives us native access to some key big data technologies such as Hadoop, HBase and Hive as well as plenty of well-supported client libraries for third-party web services (e.g. ecommerce and advertising platforms)
2. Scala is statically typed and supports straightforward data encapsulation through OOP – which is our strong preference when working with so many third-party data sources. In particular, static typing catches a whole set of data issues upfront, which saves a lot of data analysis pain later
3. Scala has functional programming features – which allows us to code in a style which has strong mathematical foundations. This proves really helpful when it comes to performing data analyses

Scala is by no means perfect (and the community can sometimes verge on the cranky side of academic), but it offers a good general purpose, ‘batteries included’ foundation for Keplar’s agile analytics projects. In future blog post series we will talk about these projects some more (we have already written a series on Mechanical Turk) – but in my next blog post I’m going to provide a brief introduction on Scala – including looking at some of the neatest features and nastiest gotchas.

To keep costs down for our clients and avoid “reinventing the wheel”, where possible we make use of existing libraries, plugins and extensions for these platforms. However this isn’t always possible, and sometimes we see an opportunity to improve (hopefully!) on the existing options.

At Keplar we’re keen to open-source any such code which we develop and own ourselves (i.e. isn’t part of a client deliverable) which we think could be useful for a wider audience. Open sourcing in-house technology has two clear benefits in our eyes: firstly it helps to support the open source ecosystems on which we depend, and secondly, the more eyes we can get on the code we use in the wild the better.

As our first tentative steps down this route, we are open sourcing a PayPal e-commerce library for CodeIgniter which we are already using in production on a couple of clients. We are releasing this library under the guise of an initiative which we are calling “Orderly” – we hope to release other e-commerce software and libraries under this banner in the future.

The library is called codeigniter-paypal-ipn, and is already available for download on GitHub. The library is designed to make it easier for developers using CodeIgniter to receive, validate and store instant payment notifications (IPNs) sent by PayPal when an order has been paid for by a customer.

The library was inspired by an earlier CodeIgniter PayPal library – Ran Aroussi’s PayPal_Lib. Compared to Ran’s library, our library adds some additional validation to your website’s interaction with PayPal, and also it adds functionality to log your orders to the database. (The launch version of codeigniter-paypal-ipn uses Doctrine 1.2 to save the orders, but we’re planning a future version which should work without Doctrine.)

For instructions on installing the library, please see the codeigniter-paypal-ipn readme file. Once the library is installed, using it in a CodeIgniter controller is quite straightforward – just write a controller like this:

// To handle the IPN post made by PayPal (uses the Paypal_Lib library).
    function ipn()
    {
        $this->load->library('PayPal_IPN'); // Load the library

        // Try to get the IPN data.
        if ($this->paypal_ipn->validateIPN())
        {
            // Succeeded, now let's extract the order
            $this->paypal_ipn->extractOrder();

            // And we save the order now.
            $this->paypal_ipn->saveOrder();

            // Now let's check what the payment status is and act accordingly
            if ($this->paypal_ipn->orderStatus == PayPal_IPN::PAID)
            {
                // Configure to send HTML emails.
                $this->load->library('email');
                $mail_config['mailtype'] = 'html';
                $this->email->initialize($mail_config);

                // Prepare the variables to populate the email template:
                $data = $this->paypal_ipn->order;
                $data['items'] = $this->paypal_ipn->orderItems;

                // Now construct the email (create a template using Smarty or similar)
                $emailBody = $this->smarty->view('ecommerce/conf_email.tpl', $data, TRUE);

                // Finish configuring email contents and send.
                $this->email->to($data['payer_email'], ($data['first_name'] . ' ' . $data['first_name']));
                $this->email->bcc('sales@CHANGEME.com');
                $this->email->from('support@CHANGEME.com', 'CHANGEME');
                $this->email->subject('Order confirmation');
                $this->email->message($emailBody);
                $this->email->send();
            }
        }
        else // Just redirect to the root URL
        {
            $this->load->helper('url');
            redirect('/', 'refresh');
        }
    }

That should be everything! Let us know how you get on with the library in the blog comments below – or over on GitHub if you prefer. We hope you find it useful, and we look forward to making more “Orderly” releases soon.

And so for the second in our series of technical blog posts, we are going to look at the opportunities to enhance your e-commerce site using geo-IP location. Geo-IP location sounds complicated but it is simply the process of determining where your individual website visitors are geographically located in the world; this is achieved by looking up each visitor’s IP address in a database which maps known IP addresses to individual countries or even cities.

As an online retailer, knowing where your website visitors are located allows you to provide them with a much more personalised shopping experience – for example, you could:

• Show specific contact details for your visitor’s country
• Price your product catalogue in your customer’s local currency
• Automatically calculate delivery times and costs for their order

These sorts of personalisations work in two ways to improve your bottom-line: firstly, they increase the level of confidence and trust which a visitor feels in your site by showing that you can treat them as a ‘local’. And secondly, they reduce friction in the check-out process, removing difficult steps for the user such as converting the given currency into their own money. Using these techniques can significantly increase conversions among overseas visitors, as we have seen above.

On to the technology: although there are various providers of geo-IP address databases, we use MaxMind because it is free, simple to use and regularly updated. Also note that many e-commerce packages such as Magento or Prestashop have MaxMind integrations available already for free or low cost – check online to see if your e-commerce package has one too.

For this example we will be proceed as if we are integrating MaxMind directly with a simple PHP-based online shop; we will use MaxMind to display some simple internationalised information to your site visitor. In future blog posts we will explore some more sophisticated localisation approaches, to really drive more sales.

Now on to the technical steps…

The first step is to install the MaxMind API and database. The commands below all assume that you are working in a web root directory, in a Debian/Ubuntu-like environment:

mkdir MaxMind
cd MaxMind
wget -r --no-parent --reject "index.html*" -nH --cut-dirs=4 http://geolite.maxmind.com/download/geoip/api/php/
mkdir data
cd data
wget http://geolite.maxmind.com/download/geoip/database/GeoLiteCountry/GeoIP.dat.gz
tar -xvf GeoIP.dat.gz
cd ../..

We’re also going to install the country flags from the famfamfam icon set, so that we can show the user their national flag:

mkdir flags
cd flags
wget http://www.famfamfam.com/lab/icons/flags/famfamfam_flag_icons.zip
unzip famfamfam_flag_icons.zip
cd ..

Next, we write a simple helper PHP file which will be included into our shop and will make it easy to run MaxMind and lookup the user’s country:

mkdir includes
vi includes/geoip.php

And enter the text:

<?php
/**
 * This is a MaxMind helper library
 * Author: Alex Dean (@alexatkeplar http://www.keplarllp.com)
 */

// Include the required PHP file
require_once dirname(__FILE__) . '/../MaxMind/geoip.inc';

// Get the country code using MaxMind geo-IP lookup
function getCountryCode() {
    $mm = geoip_open(dirname(__FILE__) . "/../MaxMind/data/GeoIP.dat", GEOIP_STANDARD);
    $ip = isset($_SERVER['HTTP_X_FORWARDED_FOR']) ? $_SERVER['HTTP_X_FORWARDED_FOR'] : $_SERVER['REMOTE_ADDR'];
    $countryCode = geoip_country_code_by_addr($mm, $ip);
    geoip_close($mm);

    return $countryCode; // 'GB', 'US' etc, or null if not found
}

With this done, let’s next display the user’s current location, so that they know that this online shop is localised for their specific country. Start by creating an index file:

vi ../index.php

And add in the following code:

<?php
require_once("includes/geoip.php");

$countryCode = getCountryCode();
if (isset($countryCode)) {
    echo "Hello, you are from <img src='flags/png/" . strtolower($countryCode) . ".png'>";
} else {
    echo "Sorry, we don't know what country you are from";
}

A few things to note about this code:

1. We need to check that MaxMind successfully found the IP address, because some IP addresses don’t exist in the MaxMind database.
2. Rather than just displaying the visitor’s country’s flag, we could use the Zend Framework to map the country code onto the country’s name. (Installing the Zend Framework is out of scope for this blog post).
3. The MaxMind database is regularly updated (typically once a month) with new and changing IP addresses, so it would be worth setting up a cronjob to update the database automatically

With the basic country-detection code functioning, the next steps would be to layer in more country-specific features, such as pricing in local currency and country-specific contact details. Also it’s a good idea to allow the user to change their country manually, in case MaxMind got it wrong or their country could not be determined. Let us know in the comments what aspects of this internationalisation you would like us to tackle next!

I hope the above is useful and leaves you with a better understanding of what geo-IP location is, and why it is such a powerful tool for e-commerce. And do let me know how you get on with the code samples – I will reply to any questions in the comments.

Are you interested in internationalising your e-commerce site? Keplar can provide you with strategic and technical support – please drop us an email to find out more.

How is web scraping useful for a business that isn’t Google-sized?  Web scraping can be used to collect and structure competitor data, making it an incredibly powerful marketing intelligence tool.  Consider online retail: using a web scraper it is possible for a retailer to automatically survey the range of products offered by competitor sites and the price each product is offered at. Because web scrapers can be automated, they can be programmed to run regularly – so companies can analyse how sensitive their own sales volumes are not just as a function of the item price, but as a function of the prices competitors price them at. It is even possible to use the data from web scrapers to dynamically price items in an online shop so that they are always competitive. If you’re an online retailer, you are quite possibly already being regularly scraped by a competitor.

In this post, we provide an example of a simple web scraper built using Groovy. We chose Groovy because it’s a powerful, agile scripting language which is great at navigating/analysing HTML. The target of our scraper is a simple test “shop” which we have setup in Keplar Labs. You are welcome to run this scraper against our test shop – please note that scraping other sites may be against their terms and conditions or even in some cases an offence. Please seek legal advice before running any scraper on someone else’s website.

Without further ado, here is the Groovy code:

/* Intelbot written by Alex Dean on 15 Jan 2010 with dependencies:
 *  - NekoHTML parser (latest stable), http://nekohtml.sourceforge.net/index.html
 *  - Xerces (2.0.0 or higher), http://www.apache.org/dist/xerces/j/
 */

// Define the pages which contain links to products - our "seeds" in crawl parlance.
def seeds = ["http://labs.keplarllp.com/shop"]

// Load the NekoHTML parser with Xerces - this lets us parse the HTML.
slurper = new XmlSlurper(new org.cyberneko.html.parsers.SAXParser())

// Now let's loop through each seed URL in turn.
seeds.each() {

	println "Accessing seed URL ${it}"
	def seedURL = new URL(it)

	seedURL.withReader { seedReader ->
		def seedHTML = slurper.parse(seedReader)

		// Show the title of the seed page we're parsing.
		println "Seed page title is ${seedHTML.depthFirst().grep{ it.name() == 'TITLE'}}"

		// Now loop through and find all the product links on this page.
		// For our purposes, a product link is any A tag inside a box div on the page.
		def productLinks = seedHTML.depthFirst().grep{ it.name() == 'DIV' && it.@class == 'box' }.collect { it.A.'@href'.toString() }
		productLinks.each {

			println "  Accessing product URL ${it}"
			def productURL = new URI(seedURL.toString()).resolve(it).toURL()

			productURL.withReader { productReader ->
				def productHTML = slurper.parse(productReader)

				// Now display the product name.
				println "    Name is ${productHTML.depthFirst().grep{ it.name() == 'H1'}}"

				// Now display the product description.
				println "    Description is ${productHTML.depthFirst().grep{ it.name() == 'P' && it.@class == 'ProductDescription'}}"

				// Now need to grab the product price.
				println "    Price is ${productHTML.depthFirst().grep{ it.name() == 'P' && it.@class == 'ProductPrice'}}"
			}
		}
	}
}

The code should be fairly self-documenting: essentially there is a loop to process each “seed page” (we have only one such page), and then an inner loop to process each product page found on a given seed page.

When you run the script in the Groovy Console, you should see output like this:

Accessing seed URL http://labs.keplarllp.com/shop
Seed page title is [Keplar Labs]
  Accessing product URL http://labs.keplarllp.com/shop/product1.html
    Name is [Flight to Dubai]
    Description is [Economy class flight to Dubai with Emirates]
    Price is [£699.00]
  Accessing product URL http://labs.keplarllp.com/shop/product2.html
    Name is [Train to Paris]
    Description is [First class train ticket to Paris on Eurostar]
    Price is [£225.50]
  Accessing product URL http://labs.keplarllp.com/shop/product3.html
    Name is [Flight to New York]
    Description is [Economy class flight to New York on United]
    Price is [£375.00]

As you can see, the scraper has successfully identified three products linked from the seed page, then accessed each of these product pages and retrieved the key information about each product (its name, description and current price). A more sophisticated scraper would simply build on this functionality, for example adding error handling and potentially adding some caching.

Web scraping is an incredibly powerful tool that has already transformed the web (by enabling services like Google search) but we believe it has the power to transform regular businesses too – once they learn to employ it. It’s also cheap to perform, making it a cost effective tool. So do try out the above code on your own computer – and let me know how you get on in the comments.

Interested in having a web scraper designed and built for your company? Send us an email to find out how Keplar can help.