Comments on: Better competitive intelligence through scraping with Groovy http://www.keplarllp.com/blog/2010/01/better-competitive-intelligence-through-scraping-with-groovy Blogging from the team at Keplar LLP Tue, 20 Sep 2011 14:56:03 +0000 hourly 1 http://wordpress.org/?v=3.2.1 By: Alex http://www.keplarllp.com/blog/2010/01/better-competitive-intelligence-through-scraping-with-groovy/comment-page-1#comment-905 Alex Fri, 19 Mar 2010 01:29:02 +0000 http://www.keplarllp.com/blog/?p=198#comment-905 Thanks for your comment! You're right, the code isn't coping with relative URLs. I've updated the offending line to be: [sourcecode language="groovy" light="true"] def productURL = new URI(seedURL.toString()).resolve(it).toURL() [/sourcecode] Hopefully that fixes it. I hadn't heard of <a href="http://groovy.codehaus.org/Grape" rel="nofollow">Grape</a> - it looks cool, will check it out! Thanks for your comment! You’re right, the code isn’t coping with relative URLs. I’ve updated the offending line to be:

def productURL = new URI(seedURL.toString()).resolve(it).toURL()

Hopefully that fixes it. I hadn’t heard of Grape – it looks cool, will check it out!

]]> By: Guy http://www.keplarllp.com/blog/2010/01/better-competitive-intelligence-through-scraping-with-groovy/comment-page-1#comment-903 Guy Tue, 16 Mar 2010 23:11:24 +0000 http://www.keplarllp.com/blog/?p=198#comment-903 Thanks, great post! First run I got: Accessing product URL /shop/product1.html Caught: java.net.MalformedURLException: no protocol: /shop/product1.html at x$_run_closure1_closure2_closure6.doCall(x.groovy:32) at x$_run_closure1_closure2.doCall(x.groovy:29) at x$_run_closure1.doCall(x.groovy:20) at x.run(x.groovy:15) so I added 'http://labs.keplarllp.com/' as the root of the URL and it worked I guess relative paths support is missing. Besides, I had to download cyberneko manually, anyone knows why @Grapes( @Grab(group='nekohtml', module='nekohtml', version='0.7.6') ) doesn't work? Thanks, great post!

First run I got:

Accessing product URL /shop/product1.html
Caught: java.net.MalformedURLException: no protocol: /shop/product1.html
at x$_run_closure1_closure2_closure6.doCall(x.groovy:32)
at x$_run_closure1_closure2.doCall(x.groovy:29)
at x$_run_closure1.doCall(x.groovy:20)
at x.run(x.groovy:15)

so I added ‘http://labs.keplarllp.com/‘ as the root of the URL and it worked

I guess relative paths support is missing.

Besides, I had to download cyberneko manually, anyone knows why
@Grapes(
@Grab(group=’nekohtml’, module=’nekohtml’, version=’0.7.6′)
)
doesn’t work?

]]>