Mutation Testing is basically testing...for tests. It ensures that your tests are truly capable of detecting errors and problems with the source code. It does this by mutating the source code itself (using ext/runkit) in such a way that an error is created in the code. If your tests detect the error, all is well with the world. If your tests do not detect the error...well, you better add a new test that does.

He mentions how it compares to code coverage and goes through the entire process you'll need to get MutateMe up and running - the download (or PEAR install), creating the extension, using the command line tool and the results of an example run of the client against a few tests.

As I said in my previous entry I was thinking about how to write a Mutation Testing engine. The "braindump" was to use PHP's built in Tokenizer to break down a class file into digestible pieces which could be mutated, and then reconstructed into a mutated file. Once you allow for a working copy of the original source (let's not mutate the original!) it turned out to be a reasonable approach.

He's not releasing it just yet ("don't get too excited until I find some time to scan through it again") but he does share some sample output from his test run - one instance where all is well and the other where one mutant "escapes" and causes an exception.

Mutation Testing is like testing, for tests. The idea is actually quite simple. Mutation testing adds small changes to your source code on the assumption that changing something, will most likely break it, which in turn means at least one test/spec should fail. [...] The reason why it's useful it that it gets around the problem of being over reliant on code coverage - just because some code executes when running tests, doesn't prove the tests will detect problems with it!

He wants to create a way that doesn't rely on external dependencies and is looking at working with the tokenizer functions native to PHP. He includes two examples of the way he's thinking - an example of simple token output and a way to go "backwards" to recreate the original source code.

From Mike's update:

The most recent version committed to the mutation_testing branch is in a testable state. I would encourage anyone interested to try out the new mutation testing feature in this branch.

You can get complete information about the project - its methods and goals - from this page on the PHPUnit wiki. If you'd like to get this new code a try and see what his mutation testing is all about, here's a link to the subversion repository to grab the latest copy.

The last weeks have been busy, but now I want to take the time to finally give you an update on the status of PHPUnit. The development of PHPUnit 3.1 is progressing nicely, about the only thing missing is finishing the documentation for all the new features

Among these new features are things like:

• improvements in mock objects
• the ability for PHPUnit to write code coverage information to a test database
• some fun new template methods
• and mutation testing that's being built on the separate branch

Sebastian quotes a definition of "mutation testing" as:

Mutation Testing, or Automated Error Seeding, is an approach where the testing tool makes some change to the tested code, runs the tests, and if the tests pass displays a message saying what it changed.

The key difference is that it makes it easier to run a series of tests rather than just one at a time, aiding in more complex testing abilities. Check out this page on the PHPUnit site for more information.

I started to collect ideas for features that I would like to see in PHPUnit but currently lack the time to implement myself.

The two listed in this post are:

• Mutation testing - an approach where the testing tool makes some change to the tested code, runs the tests, and if the tests pass displays a message saying what it changed
• Web Interface for Test Result Database - output viewer for the database logging PHPUnit is now capable of.