Today I stumbled upon an interesting and reportable scenario were I had to extract information of the weekly published Drum and Bass charts provided by BBC 1Xtra. As this information currently isn't available in any consumer friendly format like for example a RSS feed, I had to go that scraping route but didn't want to hustle with a regex approach. Since version 1.6.0 the Zend_Dom_Query component has been added to the framework mainly to support functional testing of MVC applications, but it also can be used for rolling custom website scrapers in a snap. Woot, perfect match!

He includes the code for his Bbc_DnbCharts_Scraper class he's created to show how the data is pulled in (via curl) and pushed into an object to be parsed.

]]> Fri, 17 Oct 2008 14:31:34 -0500 http://www.phpdeveloper.org/news/10771 http://www.phpdeveloper.org/news/10771 On the Developer Tutorials blog today Akash Mehta looks at screen scraping using a simple DOM library to help make it easy.

In PHP, regular expressions tend to get rather messy, DOM calls can be confusing and verbose, and often the string functions just aren't enough. In this tutorial, I'll show you how to use the middle ground - the open source PHP Simple HTML DOM Parser library, which provides jQuery-grade awesomeness for easy screen scraping without messy regular expressions.

In his examples he shows how to grab the contents of a page (google.com), find and replace information in the HTML and perform a query on the data (like finding each of the news stories out of the front page of slashdot.org).

A friend of mine who shall remain nameless pointed a post out to me on the PHP DZone web site recently. Noting that the article's content was misinformed at best and downright ignorant at worst, even when examining it sheerly from the author's knowledge of PHP as a language, this friend asked that I set the author straight.

He mentions his comments on the post correcting the author on some points as well as a more "clued in" post on the xml.lt website talking about using PHP's DOM functionality instead.

In this tutorial you will learn how to build a PHP script that scrapes links from any web page.

You'll learn to use cURL, the DOM functions, XPath and a bit of MySQL to get the job done. It's nice to see that they also include a section looking at one of the more touchy aspects of web page scraping - "is it legal?"

Just a quick post to announce (albeit a little late) that the December 2007 issue of php|architect, which includes my article on web scraping. Please buy a copy, give it a read, and feel free to post comments on the forum thread for the article. I'd love to hear some reader feedback!

Other articles in the issue include "Writing an Object/Relational Mapper in PHP", "Understanding CLI" and the cover story "Going native with mysqlnd".