Mutation testing is a powerful technique to evaluate the quality of your test suite. By introducing small changes (mutations) to your code and checking if your tests catch them, you can identify weak tests and improve code coverage quality.
Understanding Mutation Testing
Traditional code coverage metrics (line coverage, branch coverage) only measure what code is executed, not whether tests effectively catch bugs. Mutation testing goes deeper by measuring whether tests would fail if the code was intentionally broken.
How Mutation Testing Works
- 1Run your test suite to ensure it passes
- 2Create mutants by making small changes to source code
- 3Run tests against each mutant
- 4If tests fail → mutant is killed (good!)
- 5If tests pass → mutant survives (bad - tests are weak)
javascript
// Original code
function calculateDiscount(price, discount) {
if (discount > 0.5) {
return price * (1 - discount);
}
return price;
}
// Mutant 1: Change comparison operator
function calculateDiscount(price, discount) {
if (discount >= 0.5) { // Mutated: > to >=
return price * (1 - discount);
}
return price;
}
// Mutant 2: Change arithmetic operator
function calculateDiscount(price, discount) {
if (discount > 0.5) {
return price * (1 + discount); // Mutated: - to +
}
return price;
}
// Mutant 3: Remove conditional
function calculateDiscount(price, discount) {
// Mutated: entire if block removed
return price;
}Stryker: Mutation Testing Framework
Installation and Setup
bash
npm install --save-dev @stryker-mutator/core @stryker-mutator/javascript-mutator @stryker-mutator/jest-runnerConfiguration
javascript
// stryker.conf.js
module.exports = {
mutate: [
'src/**/*.js',
'!src/**/*.test.js',
'!src/**/*.spec.js',
'!src/**/index.js',
],
mutator: 'javascript',
testRunner: 'jest',
reporters: ['html', 'clear-text', 'progress'],
coverageAnalysis: 'perTest',
jest: {
projectType: 'custom',
configFile: 'jest.config.js',
},
thresholds: {
high: 80,
low: 60,
break: 60
}
};Running Stryker
bash
npx stryker runCommon Mutation Operators
Arithmetic Operators
javascript
// Original
const total = price + tax;
// Mutants
const total = price - tax; // + → -
const total = price * tax; // + → *
const total = price / tax; // + → /
const total = price; // + removedLogical Operators
javascript
// Original
if (isValid && isAuthenticated) {
grantAccess();
}
// Mutants
if (isValid || isAuthenticated) { // && → ||
grantAccess();
}
if (!isValid && isAuthenticated) { // isValid → !isValid
grantAccess();
}
if (isAuthenticated) { // Remove && condition
grantAccess();
}Conditional Statements
javascript
// Original
function getGrade(score) {
if (score >= 90) return 'A';
if (score >= 80) return 'B';
if (score >= 70) return 'C';
return 'F';
}
// Mutants
function getGrade(score) {
if (score > 90) return 'A'; // >= → >
if (score <= 80) return 'B'; // >= → <=
return 'F'; // Remove conditions
}Relational Operators
javascript
// Original
if (age >= 18) {
allowAccess();
}
// Mutants
if (age > 18) { // >= → >
if (age <= 18) { // >= → <=
if (age < 18) { // >= → <
if (age === 18) { // >= → ===
if (age !== 18) { // >= → !==Writing Tests That Kill Mutants
Example: Weak Tests
javascript
// calculate.js
function calculateTotal(items) {
return items.reduce((sum, item) => sum + item.price, 0);
}
// Weak test (doesn't catch mutants)
describe('calculateTotal', () => {
it('calculates total', () => {
const items = [{ price: 10 }, { price: 20 }];
expect(calculateTotal(items)).toBe(30);
});
// Missing edge cases: empty array, zero price, negative price
});Example: Strong Tests
javascript
// Strong test (kills mutants)
describe('calculateTotal', () => {
it('calculates total of positive prices', () => {
const items = [{ price: 10 }, { price: 20 }];
expect(calculateTotal(items)).toBe(30);
});
it('handles empty array', () => {
expect(calculateTotal([])).toBe(0);
});
it('handles zero price', () => {
const items = [{ price: 0 }, { price: 10 }];
expect(calculateTotal(items)).toBe(10);
});
it('handles decimal prices', () => {
const items = [{ price: 10.5 }, { price: 20.3 }];
expect(calculateTotal(items)).toBeCloseTo(30.8);
});
});Mutation Testing in CI/CD
GitHub Actions Integration
yaml
name: Mutation Testing
on:
push:
branches: [main]
pull_request:
branches: [main]
jobs:
mutation-test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: '20'
- run: npm ci
- run: npm run mutation-test
env:
STRYKER_DASHBOARD_API_KEY: ${{ secrets.STRYKER_DASHBOARD_API_KEY }}
- name: Upload mutation report
uses: actions/upload-artifact@v3
with:
name: mutation-report
path: reports/mutation/htmlInterpreting Results
Mutation Score
Mutation score: 78.5%
Total mutants: 245
Killed: 192
Survived: 38
Timeout: 12
No coverage: 3
Breakdown by file:
- src/utils/math.js: 95.2% (40/42)
- src/components/User.js: 65.0% (13/20)
- src/services/auth.js: 80.0% (16/20)Analyzing Surviving Mutants
javascript
// Surviving mutant indicates missing test
function getUserById(users, id) {
return users.find(user => user.id === id) || null; // Mutant: removed || null
}
// Test doesn't handle "not found" case
test('finds user by id', () => {
const users = [{ id: 1, name: 'John' }];
expect(getUserById(users, 1)).toEqual({ id: 1, name: 'John' });
// Missing: test for non-existent ID
});
// Fixed test
test('returns null when user not found', () => {
const users = [{ id: 1, name: 'John' }];
expect(getUserById(users, 999)).toBeNull();
});Best Practices
1. Focus on Business Logic
javascript
// ✅ Mutate business logic
function calculateTax(amount, rate) {
return amount * rate;
}
// ❌ Don't mutate type definitions
interface User {
id: number;
name: string;
}
// ❌ Don't mutate configuration
const API_URL = 'https://api.example.com';2. Set Appropriate Thresholds
javascript
// stryker.conf.js
module.exports = {
thresholds: {
// High quality: 80%+ mutation score
high: 80,
// Medium quality: 60%+ mutation score
low: 60,
// Fail CI if below 60%
break: 60
}
};3. Incremental Mutation Testing
javascript
// Only test changed files
module.exports = {
mutate: [
'src/**/*.js',
],
tempDirName: 'stryker-tmp',
coverageAnalysis: 'perTest',
// Only test files changed in last commit
mutate: [
{ pattern: 'src/**/*.js', mutated: true },
{ pattern: 'src/**/*.test.js', mutated: false }
]
};Integrations and Tools
Jest Integration
javascript
// stryker.conf.js
module.exports = {
testRunner: 'jest',
jest: {
config: {
testEnvironment: 'node',
testMatch: ['**/*.test.js'],
collectCoverage: true
}
}
};TypeScript Support
bash
npm install --save-dev @stryker-mutator/typescriptjavascript
module.exports = {
mutator: 'typescript',
mutate: [
'src/**/*.ts',
'!src/**/*.d.ts',
'!src/**/*.test.ts'
]
};Benefits and Challenges
Benefits
- 1Identify Weak Tests: Find tests that don't catch bugs
- 2Improve Confidence: Higher mutation score = better tests
- 3Refactor Safely: Ensure tests actually test behavior
- 4Documentation: Mutants serve as test documentation
Challenges
- 1Performance: Mutation testing is slow
- 2Equivalent Mutants: Some mutants are semantically equivalent
- 3Setup Complexity: Requires configuration
- 4False Positives: Some survivors are acceptable
Conclusion
Mutation testing is a powerful technique for ensuring test quality. While it can be slow and complex to set up, the insights it provides into test effectiveness are invaluable. Start with critical business logic, set appropriate thresholds, and integrate it into your CI/CD pipeline for continuous test quality improvement.