I was involved in making a job site few days ago. During the development, I have studied how easily anyone can develop a job site using PHP (language independent in true sense) . So I decide to write a blog post about my experience and here it goes. But note that this article is not about scaling or balancing the load on your site during heavy traffic, heh heh.

He comments on the startup process surrounding this type of site and makes suggestions about something to consider for your careers site - pulling job content from other sites in two ways - screen scraping and using the job search APIs out there.

A couple of hours ago I was playing around scraping some content from a website. All was going well until suddenly I couldn't get my script to fetch meaningful content. [...] The first thing I did was stop visiting the site for 15 minutes or so and then increase the time between requests. It briefly worked again but quickly stopped.

One simple change to his user agent string in his php.ini made the problem evaporate pointing to a user agent filtering happening on the remote side. His helpful hint involves two methods - one in just PHP and the other in cURL - to change the user agent that your scripts are sending. An even better sort of solution might be some sort of rotating array that would alternate between four or five strings to make things even more random.

In this tutorial you will learn how to build a PHP script that scrapes links from any web page. You learn how to use cURL, call PHP DOM functions, use XPath and store the links in MySQL.

You'll have to have PHP5 and the cURL extension enabled on your web server to make it all work, but the code is all there ready for you to cut and paste. The application grabs the page with cURL (including the possibility to fake your user agent), parses through the HTML with the DOM and XPath functionality to grab the links and uses the MySQL methods to store them into your database.

It's amazing how just a little optimization on the part of myspace makes crawling their site so much easier. We're going to scrape the user detail (name, age, sex, etc..) from a profile, using the header info...

The script grabs the contents of the given URL, loops through, pulls out the meta tag information and uses that as a key to grab the rest of the user's information (including name, age, city, state, etc).