Mutation testing for Apex: when 75% coverage proves nothing

The test that covers everything and checks nothing

Here is a test that gives PricingService.applyDiscount 100% line coverage.

@IsTest
static void testDiscount() {
    Decimal result = PricingService.applyDiscount(100, 0.1);
    System.assertNotEquals(null, result);
}

Every line of applyDiscount executed. The test passed. The coverage report is green, and the class clears the 75% that Salesforce requires before a production deploy. Now change the method to add the discount instead of subtracting it: it returns 110 instead of 90. The test still passes. It would pass if the method returned 0, or -1, because the only thing it ever checked was that the result was not null.

Most Apex codebases contain tests like this, and not through carelessness. The platform asks for coverage, the tooling reports coverage, and coverage is what gets reviewed. Nothing asks the question that matters: if this code were wrong, would this test fail? Mutation testing asks exactly that, mechanically, for every line of the class.

What is a mutant?

A mutant is a copy of your class with one small, deliberate defect in it: a > turned into >=, a condition negated, a return result; turned into return null;. The tool generates one mutant, runs your tests against it, records the outcome and moves to the next.

• Killed. At least one test failed against the mutant. Your tests noticed the defect.
• Survived. Every test passed with the defect in place. Your tests did not notice. The tool reports the exact line and the exact change that got through.

The mutation score is the percentage of mutants killed. A score of 90% means the tests killed nine of every ten defects the tool planted. Unlike coverage, you cannot raise it by executing more lines, only by asserting more, and asserting the right things.

The six mutations renmy runtime applies

renmy runtime mutates the parsed syntax tree of the class rather than its text, so every mutant is valid Apex and compiles. There are six kinds.

• Operator replacement. if (count > 0) becomes if (count >= 0). The relational operators >, <, >=, <=, == and != are flipped. A survivor means the boundary is never exercised: no test uses a count of exactly zero.
• Condition negation. if (isActive) becomes if (!isActive). A survivor means no test takes the false branch, or takes it and asserts nothing.
• Return value change. return result; becomes return null;, or zero, an empty string or an empty list, depending on the type. A survivor is the assertNotEquals(null, result) test above.
• Statement removal. acc.Industry = industry; is deleted. A survivor means a state change no test asserts on.
• Arithmetic operator. total + discount becomes total - discount; +, -, * and / are swapped. A survivor means a calculated value is never compared with an expected number.
• Logical operator. a && b becomes a || b, and the reverse. A survivor means no test has one operand true and the other false.

A service class of a hundred lines yields dozens of mutants, because every comparison, boolean, return and assignment is a candidate.

How to run mutation testing on an Apex class

renmy mutate takes a class name, generates the mutants for that class, runs your tests against each one and reports which survived.

# Mutate a specific class
renmy mutate PricingService

# Run only the named test class against each mutant
renmy mutate PricingService --test PricingServiceTest

# Exit non-zero if the mutation score is below 80%
renmy mutate PricingService --min-score 80

# Machine-readable report
renmy mutate PricingService --json > mutation.json

The output for a class with 42 mutants:

Generating mutants for PricingService... 42 mutants
Running tests against each mutant...

Killed: 38 (90%)
Survived: 4 (10%)
Timeout: 0

Surviving mutants:
PricingService.cls:34 operator >= → > (survived)
PricingService.cls:51 return null (survived)
PricingService.cls:67 && → || (survived)
PricingService.cls:89 statement removed (survived)

Reading the survivors

Each line is a defect your suite would not catch.

• Line 34, >= became >. A boundary. If line 34 reads if (quantity >= 10), no test uses a quantity of exactly 10. Write one.
• Line 51, return null. A return value no test inspects beyond its existence.
• Line 67, && became ||. In every test both operands have the same truth value. Add a case where exactly one is true.
• Line 89, a statement removed. A side effect nothing asserts on, often a field the test never queries back.

The fix for the opening example is one line.

@IsTest
static void appliesTenPercentDiscount() {
    Decimal result = PricingService.applyDiscount(100, 0.1);
    System.assertEquals(90, result);
}

The return-value mutant now dies, because null is not 90. The arithmetic mutant dies, because 110 is not 90. Run renmy mutate PricingService again, read the shorter list, write the next assertion. Because a local run takes seconds, this is an inner loop, like fixing a failing test.

Some mutants are equivalent: they change the code without changing its behavior, so no test can kill them. renmy runtime skips the equivalent mutants it can detect; a survivor you have read and judged harmless is a legitimate result.

Why mutation testing is only practical with a local runtime

Every mutant is a full test run. For PricingService, that is 42 runs of PricingServiceTest.

Consider what that costs in an org. The org can only run code it has been given, so each mutant means deploying a deliberately broken version of the class to a sandbox and waiting for the queued test run. Call it four minutes per mutant. 42 mutants is close to three hours for one class, during which anyone sharing the sandbox is testing against your mutants. That is why Apex never had mutation testing. Having a sandbox is not the solution; the sandbox is the reason it cannot be done.

A local runtime removes the deploy from the loop. A Java developer’s tests run on the JVM on their laptop, and nothing is sent anywhere for the code to execute; renmy runtime does the same for Salesforce Apex. It interprets your .cls and .trigger files, runs SOQL and DML against an embedded PostgreSQL started as a local process, enforces governor limits, fires triggers and flows in Salesforce’s order of execution, and honors Test.startTest, @testSetup and per-test transaction isolation. A typical Apex test completes in tens of milliseconds, SOQL and DML included; the published benchmark ran 837 tests from four open-source suites in 2.4 s locally against about 1 min 46 s in a Salesforce org including the deploy, every test passing in both (method and numbers).

At that speed, 42 runs of a test class take seconds. renmy runtime also runs mutants in parallel across workers (Pro) and skips equivalent mutants. The caveat from the mutation testing page: runs are CPU-intensive and grow with the size of the test suite run against each mutant, so --test is usually the right call. A mutation score is only worth anything if the local run behaves like the org; the public parity record lists what matches and what does not.

Mutation score as a CI gate

Coverage already has gates: Salesforce’s 75% at deploy time, and --min-coverage on renmy test. --min-score gives mutation testing the same shape: below the target, renmy mutate returns a non-zero exit code and the pipeline fails the way it does for a failing test.

Mutating every class on every commit is expensive. The practical pattern is to mutate only the classes a pull request changed, on PRs to main.

# .github/workflows/mutation.yml
- name: Mutation test changed classes
  run: |
    CHANGED=$(git diff --name-only origin/main \
      | grep '.cls$' \
      | xargs -I{} basename {} .cls \
      | grep -v Test \
      | tr '\n' ' ')

    for CLASS in $CHANGED; do
      renmy mutate "$CLASS" --min-score 80
    done
  env:
    NIMBUS_LICENSE_KEY: ${{ secrets.NIMBUS_LICENSE_KEY }}

The author who touched PricingService learns in their own PR that the change left a >= untested, and the gate scales with the size of the change rather than the codebase. Treat 80 as a starting point: set the threshold where the classes you touch most often stand today, and raise it as survivors get fixed.

The runner needs no org, no JWT certificate and no scratch org pool; the license key in NIMBUS_LICENSE_KEY is the only secret. Mutation testing is a Pro feature and CI runs are part of Team; both are free during the public beta (pricing). The rest of the test stage is in CI/CD integration.

What about the 75% rule?

It stays: Salesforce requires 75% line coverage for a production deploy. The question is where mutation testing fits beside it.

Coverage comes first, because a line that never executes cannot kill a mutant either. Branch coverage comes second: an org reports line coverage only, while renmy runtime measures line and branch coverage locally, which already exposes the if whose false branch no test takes (Apex branch coverage). Mutation score comes third, and it is the only one of the three that measures assertions rather than execution. A class can hold 100% line and branch coverage and a mutation score of zero, if its tests call everything and check nothing.

For a tech lead gating on 75%: keep the gate, because it is the platform’s, and add a mutation gate on changed classes, because it checks what the coverage gate cannot. The longer argument is Apex code coverage is not a test strategy.

Who this is for

Apex developers who want to know whether their tests would catch a bug, not only whether they ran. Tech leads who gate on 75% and suspect the number is being met rather than earned. If coverage itself is still the problem, start with coverage; mutation testing builds on it.

Frequently asked questions

What is mutation testing for Apex?

Mutation testing plants small deliberate defects in an Apex class, one at a time, and runs the tests against each changed copy. A defect the tests fail on is killed; one the tests pass with has survived, and marks a test that asserts too little. The mutation score is the percentage killed. renmy runtime, formerly Nimbus, implements this for Salesforce Apex with renmy mutate, using six mutation kinds; to our knowledge it is the first mutation testing for Apex.

What is a good mutation score for Apex?

There is no platform rule for mutation score the way there is for the 75% coverage requirement. renmy mutate --min-score 80 is the documented example and a reasonable starting gate; the useful threshold is the one your frequently changed classes reach today, raised as survivors are fixed.

Does mutation testing replace code coverage?

No. Salesforce still requires 75% line coverage for a production deploy, and a line no test executes cannot kill a mutant, so coverage comes first. Mutation testing measures what coverage cannot: whether the tests that execute a line would fail if the line were wrong.

Can I run mutation testing against a Salesforce org?

Not practically. Each mutant is a full test run, and an org can only run code it has been given, so every mutant would be a deploy of a deliberately broken class followed by a queued test run, roughly four minutes each. renmy runtime runs each mutant locally against an embedded PostgreSQL in seconds.

Is mutation testing in renmy runtime free?

Mutation testing is part of the Pro tier, and running it on a CI runner is part of the Team tier. Both are free during the public beta; the list prices are $19 per month for Pro and $49 per developer per month for Team. The Free tier for individual developers is $0 with no card.