Since sometimes you just don't want all of the results of a query back at once, Rob Young has posted a solution of his own using the Iterators included with PHP as a part of the SPL. His solution is to wrap it in a ChunkedQueryIterator that handles the work behind the scenes.

When executing large queries it's usually best not to load the whole result set in one go. Memory isn't infinite and PHP isn't renowned for handling it very well. So the obvious answer is to chunk the large query in to lots of smaller queries. [...] We want something to which we can just provide a PDO object, an SQL query and the chunk size. We should then be able to iterate over the resulting object as though it were a single result set.

He includes two code snippets of it in action, but asks the question of his readers - "How do you handle large database queries?" - to get some feedback on other alternatives.

In this recent post to his blog Michael Caplan looks at a feature of PHP that's sometimes forgotten when pushing out larger chunks of data - flushing.

As a recluse who prefers hiding behind servers rather than dancing around your web browser's canvas, I was intrigued with their server side recommendations - however sparse they may be. In particular, flushing generated head content early to speed up overall page delivery and rending time was a technique new to me.

Michael looks at what "flushing generated head content" means and includes a scenario - pulling the top palettes from the COLOURlovers site - and some performance stats on page load time and response time directly from the server (complete with graphs).

On the ThinkPHP blog today, there's a quick hint about dealing with larger files both with and whithout PHP.

As one man was quoted "640K of memory should be enough for anybody" no one will need to access more than 2 GB data. What happens if you - just for scientific reasons of course - try to access larger files using your 32bit hardware and your favorite programming language PHP?

They give the example of opening a large 2 gig file with PHP and the resulting error that would pop up. They try a few differnt ways before getting down to more of a non-PHP PHP solution (yes, you read that right). They decided, instead, to create a script to work with the file chunked, using an exec() call to the unix split command to break it up.