Three ways a Salesforce pipeline fails that aren’t your code

renmy runtime was called Nimbus until October 2026. Same tool, same binary; the record of the rename is at Nimbus is now renmy runtime.

Who this is for

DevOps engineers and team leads who own a Salesforce pipeline and have re-run a red build without changing anything, and watched it pass. No prior contact with a local runtime is assumed. Salesforce CI without scratch orgs covers the whole pipeline; this article is about the failures in between.

Why does a Salesforce pipeline fail when the code is fine?

Because most of the pipeline is not running your code. It is renting the place where your code can run. An org-based Apex test job has this shape:

steps:
  - name: Decrypt the JWT key
    run: openssl enc -aes-256-cbc -d -in server.key.enc -out server.key -k $JWT_KEY_PASSPHRASE

  - name: Authenticate to the Dev Hub
    # Needs the private key, the connected app's consumer key and an integration user
    run: sf org login jwt --client-id $CONSUMER_KEY --jwt-key-file server.key --username $DEVHUB_USERNAME --set-default-dev-hub

  - name: Create a scratch org
    # Burns a slot from a finite pool; fails when the pool is exhausted
    run: sf org create scratch -f config/project-scratch-def.json -d -y 1 -a ci-scratch

  - name: Deploy source
    # 4-8 minutes, every run
    run: sf project deploy start -o ci-scratch

  - name: Run tests
    # Another 3-5 minutes
    run: sf apex run test --target-org ci-scratch --code-coverage --result-format junit --output-dir test-results --wait 20

  - name: Delete the scratch org
    # If anything above crashed, this does not run
    run: sf org delete scratch -o ci-scratch -p

One of those six steps executes Apex. The other five exist because Apex has never had a runtime outside Salesforce’s data centres: the compiler, the database and the test runner all live in the org. A Java team’s CI runs mvn test on the runner and depends on a JVM download. An Apex team’s CI depends on a certificate, a connected app, a Dev Hub allocation and a round trip to Salesforce before a single assertion executes. Total time when everything works: 12–20 minutes per run. Three causes of failure come back more often than any other.

Failure 1: an empty scratch org pool

Every CI run creates a scratch org, and every scratch org counts against the Dev Hub’s allocation: the edition caps both active orgs and daily creations. A run that never reaches the delete step, because a deploy timed out or a job was cancelled, leaves an org that holds its slot until it expires.

Release weeks mean more pushes, more pushes mean more orgs, and the pool runs dry exactly when the team most needs CI. sf org create scratch fails, and after the third retry someone merges to main hoping the tests pass in production. The cost is the hour a developer spends on their own stack trace and the pipeline log before finding an org-limit error unrelated to the pull request.

The workarounds are familiar: if: always() on the delete step, --duration-days 1 so orphans expire overnight, a nightly job that deletes whatever the Dev Hub still lists. Each shrinks the window. None closes it, because the pool is finite and the pushes are not.

Failure 2: an expired JWT certificate or connected app

The JWT bearer flow authenticates a machine to Salesforce with a certificate. Someone generated it with an expiry date when the pipeline was set up, uploaded the public half to a connected app, encrypted the private half into the repository and put the passphrase in the secrets store. Then they moved teams.

The day it expires, every job fails at the authentication step, and nothing in the repository explains why. Fixing it means a new key pair, whoever still has admin rights on the connected app, a re-encrypted private key and a rotated passphrase, with CI down for the duration. A deactivated connected app or a changed integration user fails the same way. The handshake that was never the part of CI that found bugs becomes the part that broke at 2am.

Teams work around this with a rotation calendar, a runbook and a second certificate staged before the first lapses: real operational work for a credential whose only purpose is to let a unit test run.

Failure 3: a slow or blocked network to the org

Two of the six steps are long conversations with Salesforce. The deploy pushes the source tree and waits for the org to compile it; the test run waits for the org’s test queue. On a good day that is 4–8 minutes and another 3–5. On a bad day the deploy sits in the queue, --wait runs out, and the step fails with a timeout instead of a result.

A self-hosted runner behind a corporate proxy needs the Salesforce hosts allow-listed. A hosted runner’s address changes from job to job, so an org that restricts the integration user’s login IP ranges rejects it. And when Salesforce has an incident, nothing gets tested, because the only Apex runtime the pipeline knows is on the other side of the incident.

The workarounds are longer timeouts, retries on the deploy step, and a nightly schedule when the queue gets bad. Each accepts the external dependency and manages around it.

What removes all three: Apex tests on the CI runner

The three failures share one root: the test job has to reach an org because the tests have nowhere else to run. Remove that premise and the steps the failures live in are gone.

renmy runtime is a local Salesforce Apex runtime. It is an interpreter for Apex classes, interfaces, enums, generics and triggers, for SOQL and DML, and for Salesforce Flows (autolaunched, record-triggered and screen), running against an embedded PostgreSQL on the machine where the source is. Governor limits are enforced, Test.startTest/Test.stopTest, @testSetup, per-test transaction isolation and the Stub API behave as on the platform, and triggers and flows fire in order of execution. It is one binary for macOS, Linux and Windows with no Docker, no JVM and no org; not a mock, not a transpiler. How closely it matches the platform is a public record rather than a claim: renmy.net/parity.

The job

name: Apex Tests

on:
  push:
    branches: [main]
  pull_request:

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Install renmy runtime
        run: curl -fsSL https://renmy.net/install.sh | sh

      - name: Run tests
        run: |
          renmy test "*" \
            --coverage --coverage-report coverage.xml \
            --results-xml results.xml
        env:
          NIMBUS_LICENSE_KEY: ${{ secrets.NIMBUS_LICENSE_KEY }}
          NIMBUS_PROFILE: ci

      - name: Publish test results
        uses: EnricoMi/publish-unit-test-result-action@v2
        if: always()
        with:
          files: results.xml

Total time: 30–90 seconds. The published benchmark covers the test step itself: four open-source Apex suites, 837 tests, about 1 min 46 s for deploy and run in a Salesforce org against 2.4 s locally, roughly 45× faster, every test passing in both (method and numbers). results.xml is JUnit XML, coverage.xml is Cobertura with line and branch coverage, and the exit code is non-zero when a test fails, so the merge gate you already have keeps working. The GitLab CI version and the other templates are on renmy.net/ci; the wiring guide is Apex CI without an org.

What the job depends on now

No scratch org pool. Nothing is created, so nothing can be exhausted, orphaned or expire, and the delete step is gone because there is nothing to delete.

No certificate, no connected app, no Dev Hub. The job has one secret, the licence key in NIMBUS_LICENSE_KEY, issued from the portal. There is no private key in the repository and no rotation calendar.

No network path to the org. The network is used to install the binary and to check the licence key, and the install can be cached with the rest of the runtime:

- name: Cache renmy runtime
  uses: actions/cache@v4
  with:
    path: |
      ~/.nimbus
      .nimbus/cache
    key: renmy-${{ hashFiles('force-app/**/*.cls', 'force-app/**/*.trigger') }}
    restore-keys: renmy-

A slow org, a blocked proxy or a Salesforce incident cannot reach a job that never calls Salesforce.

Where the org still belongs: the deploy job

Nothing above removes the org from the pipeline, only from the stage that runs on every push and used it as a test runner. The deploy job is different: the org is the final authority on whether the metadata compiles and whether its own tests pass.

renmy deploy is the free, gated version of that step. It snapshots the selected source into a content-addressed bundle, runs the configured local test and coverage gates against it, validates the identical bytes in the target org with a real check-only validation, and only then deploys (renmy.net/deploy). renmy release validate and renmy release deploy split the two around a human approval, with an immutable receipt (Pro, free during the public beta; renmy.net/release).

The deploy job still authenticates to Salesforce, so the connected app and its certificate still exist. What changes is how often they sit on the critical path: once per release, in a job a release manager is watching, instead of in every check a developer waits on to merge. The deploy job has one more non-code failure of its own: a pipeline that installs @salesforce/cli fresh on every run gets a different CLI whenever Salesforce publishes one. renmy toolchain sf install --version … pins the exact version (Pin the Salesforce CLI version in CI).

“We already have a CI sandbox” is the usual objection, and the answer is that you keep it, for what only an org can do: Lightning pages, permission checks, integrations against real endpoints, the final validation of the payload. The comparison is at renmy runtime vs scratch orgs; what a scratch org costs per day is priced in The scratch org tax.

How to adopt it without a flag day

Add the renmy runtime job beside the existing one and leave both running for a few weeks. Watch which one finds bugs first and which one goes red for reasons unrelated to the pull request. Then promote the renmy runtime job to the required check and demote the scratch org workflow to a nightly platform-fidelity gate.

CI runs are part of the Team tier, with a CI runner licence, seats and the self-hosted assurance console. Team is free during the public beta; the list price is $49 per developer per month. The Free tier, for individual developers at $0 with no card, covers unlimited local test runs and gated deploys on one machine (pricing). The calculator on renmy.net/teams has a line for CI failures unrelated to the code; fill in your own numbers.

Try it

curl -fsSL https://renmy.net/install.sh | sh          # macOS, Linux
irm https://renmy.net/install.ps1 | iex              # Windows PowerShell
cd your-sfdx-project
renmy test "*" --coverage

The quickstart takes you from install to a first green run on your own machine, with the same command the runner will execute. Issues, docs, examples and the released binaries are at github.com/renmy-solutions/renmy; the CI templates are on renmy.net/ci; questions go to the Slack.

Frequently asked questions

Why does a Salesforce CI pipeline fail when no code changed?

Because an org-based pipeline depends on things the code does not: a scratch org from a finite Dev Hub pool, a JWT certificate and connected app, and a network path to Salesforce. An empty pool, an expired certificate or a slow org fails the job before any Apex executes. A re-run that passes without a change is the signature of an environment failure, not a code failure.

Do I still need the JWT certificate and connected app?

For the deploy job, yes: metadata has to reach the target org, and renmy deploy or renmy release deploy authenticates to it through the Salesforce CLI. For the test job, no. renmy runtime runs the Apex tests on the CI runner against an embedded PostgreSQL, so that job needs no org credentials, only the licence key in NIMBUS_LICENSE_KEY.

What does the renmy runtime test job need from the network?

The install of the binary and a check of the licence key. There is no call to a Salesforce org, so a slow deploy queue, a blocked proxy or a Salesforce incident does not affect the job. The binary and .nimbus/cache can be cached between runs.

Are Apex tests that never touch an org trustworthy?

renmy runtime executes real Apex, SOQL, DML and Salesforce Flows with governor limits enforced, per-test transaction isolation and the platform’s order of execution; it is not a mock and not a transpiler. How closely it matches the org is recorded publicly at renmy.net/parity rather than claimed. The org’s own compile and test run still happen inside the check-only validation of renmy deploy, so the final word stays with Salesforce.