Beware Mutants
Testing Your Tests
Ian Littman / @iansltx
AustinPHP / February 11, 2016
whoami
- I write web apps, primarily APIs, for a few companies
- I like conferences
- Attended SunshinePHP 2016 last weekend
- Attending Lone Star PHP 2016 in April
- Speaking at php[tek] 2016 in May
- (@)iansltx everywhere
So, what's this then?
- The problem with code coverage (why mutation testing?)
- What is mutation testing?
- How do you mutation test PHP code? (demo!)
Who watches the watchers?
- Tests are code
- Failures on boundary conditions
- Outright bugs
- Incomplete
- Unit tests should catch common and edge cases during original dev, and catch regressions later
- Code coverage indicates how much of a code base is executed when running a test suite...
- ...but it can be misleading
100% code coverage
The solution: mutation testing
- Tweak your code (simulate regressions/typos)
- Run your test suite
- See what happens!
tweak your code
- Flip booleans
- Off-by-one
- Modify equalities
- Negate expressions
- Change return values
- Change math operators
- Change logical operators
- ...and more!
Run your test suite
- Line coverage is a starting point
- You're probably going to miss a mutation if it's not in covered code
- High coverage -> more chances to catch mutants...
- ...or watch them escape
- Run these tests in a secure location!
- Mutation testing takes awhile
- Runs all of, or a part of, your test suite for each mutation
see what happens
- Tests fail (good!) <- mutant killed
- Tests time out (infinite loop; okay!)
- Tests fatal (probably fine)
- Test suite passes (probably bad) <- mutant escaped
- "Probably" due to false positives
- Fatals and timeouts are fine because you'll catch those sorts of regressions by running your automated test suite, rather than in production
How do we use this?
-
PiTestMutagenesisMutateMeHumbug - from the maker of Mockery (Pádraic Brady)
- Ties in with PHPUnit
- Current version of 4.8 fatals
- Some versions of 4.x work fine
- Current version of 5.x (5.2.4) works fine
- JSON config
- Override/exclude code or test dirs
- Set timeouts
How do we use Humbug?
-
PHP 5.4+
-
composer global require 'humbug/humbug=~1.0@dev'
-
Add ~/.composer/vendor/bin to $PATH if you haven't already
-
humbug configure
-
humbug
- Wait for results (you'll see mutants get killed/escape/time out/fatal as test suites are run)
We've got results!
- Summary
- # killed/escaped/timed out/errored/not covered
- % of mutants killed (Mutation Score Indicator)
- % of mutants in covered code
- % of mutants killed in covered code
- Text log
- Summary data, plus...
- For each escaped mutant
- Location of code diff
- Code diff
- Mutation type (e.g. conditional boundary change)
- JSON log (more detailed than text log)
- Tests executed
- Class/method/line/test output
Demo Time!
Questions?
Beware Mutants: Testing Your Tests - AustinPHP February 2016
By Ian Littman
Beware Mutants: Testing Your Tests - AustinPHP February 2016
- 1,824