News Feed
Sections




News Archive
feed this:

Looking for more information on how to do PHP the right way? Check out PHP: The Right Way

SitePoint PHP Blog:
Crawling and Searching Entire Domains with Diffbot
July 02, 2015 @ 09:41:39

The SitePoint PHP blog has a new tutorial posted, the first part in a new series, showing you how to create a "powerful custom search engine" with the help of the Diffbot service. In this first part they help you get everything you need set up (including a VM to run it from).

In this tutorial, I'll show you how to build a custom SitePoint search engine that far outdoes anything WordPress could ever put out. We'll be using Diffbot as a service to extract structured data from SitePoint automatically, and this matching API client to do both the searching and crawling. I'll also be using my trusty Homestead Improved environment for a clean project, so I can experiment in a VM that's dedicated to this project and this project alone.

He walks you through each step of the process, first creating the "crawljob" script and then executing it to gather the results. He also shows how to show this information via a simple GUI when searches are performed. A Diffbot PHP client library makes creating the crawljob simpler and lets you configure things like max number of items to crawl, patterns to match and what URLs to follow on the pages. Running the script creates the job which is then executed immediately. The same library makes search the data simpler too, using a "search" method along with some special tagging, and returning a JSON result with the matching records.

0 comments voice your opinion now!
crawl domain diffbot search engine part1 series tutorial

Link: http://www.sitepoint.com/crawling-searching-entire-domains-diffbot/

Debuggable Blog:
Crawl Google, they do the same to you
June 11, 2008 @ 10:23:07

On the Debuggable blog, Felix Geisendorfer has posted some code (thought up by Marc Grabaniski) to go through Google and find the pages that they have indexed for your site. Their goal is to check and see if the migration of a site was successful.

Just get a list of all pages google has indexed from your site and then use that as your basis for checking if your migration worked or not. This is very convenient because you do not have to know all your own urls yourself, and you'll only get the relevant ones (if they are not in google they are unlikely to have traffic).

The code is included as well as an example usage. He also points out FixtureShell for more command-line CakePHP examples.

0 comments voice your opinion now!
crawl google migration success link cakephp framework



Community Events

Don't see your event here?
Let us know!


language conference yii2 series project api php7 example list opinion part2 application introduction composer framework podcast symfony interview community laravel

All content copyright, 2015 PHPDeveloper.org :: info@phpdeveloper.org - Powered by the Solar PHP Framework