Apex tests in Azure DevOps without a Salesforce org

renmy runtime was called Nimbus until October 2026. Same tool, same binary; the record of the rename is at Nimbus is now renmy runtime.

Who this is for

DevOps engineers and release managers who run Salesforce on Azure DevOps: Azure Pipelines for the build, a branch policy on main, Environments for production. No prior contact with a local runtime is assumed. The GitHub Actions and GitLab CI versions of this article are Apex tests in GitHub Actions without a Salesforce org and Apex tests in GitLab CI without a scratch org.

What does an Azure Pipelines job for Apex tests need today?

The stage that runs Apex tests has to borrow a Salesforce org, because until now there has been nowhere else for Apex to run: a JWT certificate in Secure files, sf org login jwt against a Dev Hub, sf org create scratch from a finite pool, sf project deploy start, sf apex run test, and a delete step that does not run when something above it crashed. renmy.net/ci puts the total at 12 to 20 minutes per run, and the stage fails for reasons that have nothing to do with the pull request: the Dev Hub is out of scratch orgs, the certificate expired, the network to the org is slow.

What is a local Apex runtime, and why does it change the CI job?

In every other ecosystem the tests run where the code is. A Java team’s pipeline runs mvn test on the agent: the JVM is a download, the database is a container, nothing outside the agent is involved. Apex never had that option. The compiler, the runtime, the database and the test runner live in Salesforce’s data centres, so the only way to run an Apex test has been to put the code in an org and ask the org to run it.

renmy runtime is the missing piece: a runtime for Salesforce Apex that runs on a developer’s machine or a CI agent. It executes Apex classes, interfaces, enums, generics and triggers, SOQL and DML, and Salesforce Flows against an embedded PostgreSQL, with the platform’s rules in force: governor limits, Test.startTest/Test.stopTest, @testSetup, per-test transaction isolation, the Stub API, triggers and flows in order of execution. One binary for macOS, Linux and Windows, no Docker, no JVM, no org. Not a mock framework, not a transpiler; the Apex runs as Apex. How closely it matches the platform is a public record: renmy.net/parity. The published benchmark runs four open-source suites, 837 tests, in about 1 min 46 s in a Salesforce org (deploy and run) against 2.4 s locally, roughly 45× faster, every test passing in both: renmy.net/benchmarks.

For an Azure Pipelines job the test stage becomes what it is for Java or Python: install a binary, run the test command, publish two files.

Apex tests in Azure Pipelines: the complete azure-pipelines.yml

The pipeline

# azure-pipelines.yml
trigger:
  branches:
    include: [main]
# Pull requests: a build validation branch policy on main (Azure Repos),
# or a pr: trigger when the repository is hosted on GitHub.

pool:
  vmImage: ubuntu-latest

steps:
  - checkout: self

  - script: curl -fsSL https://renmy.net/install.sh | sh
    displayName: Install renmy runtime

  - script: |
      renmy test "*" \
        --coverage --coverage-report coverage.xml \
        --results-xml results.xml \
        --profile ci
    displayName: Run Apex tests
    env:
      # A secret pipeline variable. Azure does not hand secrets to scripts
      # unless they are mapped here. CI runs are part of the Team tier.
      NIMBUS_LICENSE_KEY: $(NIMBUS_LICENSE_KEY)

  - task: PublishTestResults@2
    displayName: Publish test results
    condition: succeededOrFailed()
    inputs:
      testResultsFormat: JUnit
      testResultsFiles: results.xml
      testRunTitle: Apex tests

  - task: PublishCodeCoverageResults@2
    displayName: Publish coverage
    condition: succeededOrFailed()
    inputs:
      summaryFileLocation: coverage.xml

What each step does

Install. One curl, one binary. No JVM, no Docker, no service container. renmy runtime starts its own embedded PostgreSQL when the test command begins and tears it down when the job finishes.

Run. renmy test "*" runs every test class. --results-xml results.xml writes JUnit XML. --coverage --coverage-report coverage.xml writes Cobertura XML; the extension chooses the format, so coverage.html and coverage.json are the other two. --profile ci activates the CI overrides from renmy.properties, described below. The exit code is non-zero when any test fails, so the step fails, the stage fails, and a build validation policy on main blocks the pull request.

The secret. There is one: the licence key for CI runs, from the portal, stored as a secret pipeline variable. CI runs are part of the Team tier, free during the public beta (pricing). Azure does not expose secret variables to scripts as environment variables, hence the env: mapping. No JWT certificate, consumer key or Dev Hub username appears in this stage.

Publish. PublishTestResults@2 reads the JUnit file into the Tests tab: every method with its duration, every failure with its message. PublishCodeCoverageResults@2 reads the Cobertura file into the Code Coverage tab; the same file goes to SonarQube or Codecov unchanged. condition: succeededOrFailed() makes both tasks run when the test step failed, which is when you want them.

The task names, condition, env and the secret mapping are Azure’s. Everything that starts with renmy is identical on GitHub Actions, GitLab CI, Jenkins or a plain shell: one install, one test command, two files out; the GitHub Actions and GitLab CI wiring is in Apex CI without a connected org. Total time for the stage, per renmy.net/ci: 30 to 90 seconds, with the network used only to download the binary and check the licence key.

The CI profile in renmy.properties

renmy runtime reads Quarkus-style configuration profiles from a renmy.properties file committed to the repository. Keys prefixed with %ci. apply when --profile ci is on the command or NIMBUS_PROFILE=ci is in the environment.

# renmy.properties, committed to the repository

# Defaults: relaxed for local development
renmy.governor.mode=warn
renmy.trace.enabled=true

# CI overrides, active with --profile ci or NIMBUS_PROFILE=ci
%ci.renmy.governor.mode=strict
%ci.renmy.test.parallel=4
%ci.renmy.trace.enabled=false

The agent gets strict governor limit enforcement, four parallel workers and no trace output, defined once in source control rather than spread across pipeline variables; a laptop gets the relaxed defaults from the same file. Key reference: renmy.net/configuration.

Faster runs: caching and sharding on Azure Pipelines

renmy runtime caches parsed Apex in .nimbus/cache/; on a project with hundreds of classes that cuts start-up from several seconds to under a second. Keep the directory between runs with Cache@2, keyed on the Apex source and placed before the test step.

  - task: Cache@2
    displayName: Cache parsed Apex
    inputs:
      key: 'renmy | "$(Agent.OS)" | force-app/**/*.cls, force-app/**/*.trigger'
      restoreKeys: |
        renmy | "$(Agent.OS)"
      path: $(Build.SourcesDirectory)/.nimbus/cache

A suite that takes minutes even locally can be split across agents with --shard N/M: a strategy: matrix with a SHARD variable per entry runs renmy test "*" --shard $(SHARD)/4 --results-xml results-$(SHARD).xml concurrently. Classes are balanced by recorded duration from .nimbus/test-durations.json (commit it or cache it; without it the split is even by class name), all methods of a class stay in one shard so @testSetup behaves, and each shard’s JUnit XML carries a shard="N/M" attribute. Details: Shard an Apex test suite across CI jobs.

Where the org comes back: validate, approve, deploy

“Tests passed” and “deploy succeeded” are different claims. The org still has to accept the payload, and in a regulated or multi-environment team a person decides in between, hours later. renmy release is built for that gap: renmy release validate proves a payload locally and then in the target org with a real Salesforce check-only validation, and writes an immutable receipt; renmy release deploy accepts only that receipt. Approval sits between the two.

The validate and deploy stages

# azure-pipelines.yml, as stages. The test job above becomes the first stage.
trigger:
  branches:
    include: [main]

pool:
  vmImage: ubuntu-latest

variables:
  # Non-secret variables reach scripts as environment variables.
  SF_VERSION: '<pinned Salesforce CLI version>'

stages:
  - stage: Test
    jobs:
      - job: ApexTests
        steps:
          # the steps from the first pipeline

  - stage: Validate
    dependsOn: Test
    condition: and(succeeded(), eq(variables['Build.SourceBranch'], 'refs/heads/main'))
    jobs:
      - job: ValidateRelease
        steps:
          - checkout: self
          - script: curl -fsSL https://renmy.net/install.sh | sh
            displayName: Install renmy runtime
          - script: renmy toolchain sf install --version "$SF_VERSION"
            displayName: Pin the Salesforce CLI
          # Authenticate to the target org with the Salesforce CLI here, as your
          # deploy job does today. renmy sf <command> passes it through unchanged.
          - script: renmy release validate --release-profile ci
            displayName: Validate the release
            env:
              NIMBUS_LICENSE_KEY: $(NIMBUS_LICENSE_KEY)
          - task: PublishPipelineArtifact@1
            displayName: Publish receipt and bundle
            inputs:
              targetPath: .nimbus/releases
              artifact: release

  - stage: Deploy
    dependsOn: Validate
    jobs:
      - deployment: DeployToProduction
        # Approvals and checks are configured on the Environment, not in YAML.
        environment: salesforce-production
        strategy:
          runOnce:
            deploy:
              steps:
                # A deployment job does not check out source by default, and the
                # deploy needs sfdx-project.json and its packageDirectories on disk.
                - checkout: self
                - download: none
                - task: DownloadPipelineArtifact@2
                  displayName: Download receipt and bundle
                  inputs:
                    artifact: release
                    path: $(Build.SourcesDirectory)/.nimbus/releases
                - script: curl -fsSL https://renmy.net/install.sh | sh
                  displayName: Install renmy runtime
                - script: renmy toolchain sf install --version "$SF_VERSION"
                  displayName: Pin the Salesforce CLI
                # Authenticate to the target org with the Salesforce CLI, as above.
                - script: |
                    renmy release deploy \
                      --receipt .nimbus/releases/rel_*.json \
                      --release-profile ci \
                      --confirm-production
                  displayName: Deploy the validated payload
                  env:
                    NIMBUS_LICENSE_KEY: $(NIMBUS_LICENSE_KEY)

Three details carry the design. The org credentials appear in the Validate and Deploy stages and nowhere else, so the JWT certificate that used to be needed on every push is needed on merges to main. renmy toolchain sf install --version pins the Salesforce CLI in both stages, so the agent that validated and the agent that deploys provably run the same toolchain (Pin the Salesforce CLI version in CI). And the deployment job runs checkout: self, because the Salesforce CLI deploys in project context: the checkout must contain sfdx-project.json and every path its packageDirectories declares; an empty force-app/ is fine. The artifact holds one receipt, so the glob in --receipt resolves to one file. The release profile keys and the secrets are in the release-in-CI reference under renmy.net/docs/ship.

What the Environment approval does, and what --confirm-production does not

The approval lives on the Environment: under Pipelines, Environments, salesforce-production, Approvals and checks, add the approvers. A deployment job that targets the Environment waits there before its first step, and the receipt and bundle wait with it as an ordinary pipeline artifact. Receipts carry no org aliases, no credentials and no source, so publishing one between stages leaks nothing.

--confirm-production is not that approval. It is the non-interactive acknowledgement that the gate was already passed, because a hosted agent has no terminal to type a confirmation into. A production profile can additionally require a counter-signature, so the identity that validated a release cannot be the identity that ships it.

Before renmy release deploy sends a byte it re-checks the receipt: org fingerprint (one-way, never an alias), bundle digest, receipt age against the 10-day Salesforce quick-deploy window, toolchain version policy, and whether baseline components drifted in the org since validation. Where Salesforce supports it, the validated org-side job is quick-deployed with no source retransmitted; otherwise the exact stored bundle is re-sent. No gates rerun, no source is restaged, so the bytes cannot change between the proof and the deploy. A changed, expired or wrong-org receipt is rejected; there is no --force and no --skip-verify. Each field’s reasoning: What a release receipt should contain.

Exit codes the pipeline can branch on

Each failure mode has its own exit code, so a follow-up step can react without parsing prose: 21 a local gate failed, 22 Salesforce rejected the payload, 23 the deploy failed, 25 a release integrity violation, 27 baseline components drifted in the org after validation, 29 a required counter-signature is missing. For a job stuck in the org’s deploy queue, renmy release status, watch and requeue are the runbook: Fix a stuck Salesforce deployment.

renmy release is Pro, free during the public beta, no card. When one person decides and ships in the same minute, the Free renmy deploy is the right shape: local gates, a check-only validation of the same bytes in the org, then the deploy, in one command. renmy.net/deploy and renmy.net/release cover both.

What still needs an org

Lightning pages, permission checks, integration tests against real endpoints and the final validation of the payload belong in the org, and the Validate stage keeps that last one there on purpose. What moves to the agent is the part that runs on every push: the Apex tests. “I already have a CI sandbox” is the usual objection, and the answer is that you keep it. Add the renmy runtime stage beside the existing pipeline, watch which one finds bugs first and which one breaks for unrelated reasons, then make the renmy runtime stage the required check and move the scratch-org pipeline to a nightly run. In full: renmy runtime vs scratch orgs.

Try it

curl -fsSL https://renmy.net/install.sh | sh          # macOS, Linux
irm https://renmy.net/install.ps1 | iex              # Windows PowerShell
cd your-sfdx-project
renmy test "*" --coverage

The quickstart takes you from install to a first green run on your own machine before you touch the pipeline. Issues, docs, examples and the released binaries are at github.com/renmy-solutions/renmy; questions go to the Slack.

Frequently asked questions

Does Azure Pipelines need a Salesforce org to run Apex tests with renmy runtime?

No. renmy runtime executes the Apex tests on the Microsoft-hosted agent against an embedded PostgreSQL that it starts and stops itself, so the test stage needs no org, no scratch org, no Dev Hub and no JWT certificate. The only secret is the licence key in NIMBUS_LICENSE_KEY. The target org is needed again in the deploy stage, where renmy release validate runs a real check-only validation before anything ships.

Which Azure DevOps tasks read the output of renmy test?

PublishTestResults@2 with testResultsFormat: JUnit reads the file written by --results-xml, and PublishCodeCoverageResults@2 reads the Cobertura file written by --coverage --coverage-report coverage.xml. Both are standard Azure tasks with no plugin involved. The task names are Azure’s; the renmy commands and their output files are the same on GitHub Actions, GitLab CI and any other runner.

How do I put a human approval between validation and the production deploy?

Run renmy release validate in one stage and publish .nimbus/releases/ as a pipeline artifact. Create an Azure Environment with Approvals and checks, and run renmy release deploy in a deployment job that targets it; the job waits for the approver before its first step. The deploy passes --receipt and --confirm-production, the non-interactive acknowledgement that the gate was passed, not the approval itself. The receipt is re-checked for org fingerprint, bundle digest, age and drift before any byte is sent.

What does it cost to run renmy runtime on Azure Pipelines?

CI runs are part of the Team tier, which is free during the public beta; the list price is $49 per developer per month. The JUnit and Cobertura XML outputs are Pro formats and are included. The Free tier covers unlimited local test runs and gated deploys for an individual developer on one machine, but does not run tests in CI.

Can I keep the scratch-org pipeline while I try this?

Yes, and that is the recommended way in. Add the renmy runtime stage beside the existing one and watch which finds bugs first and which fails for reasons unrelated to the code. Then make the renmy runtime stage the required check in the branch policy and move the scratch-org pipeline to a nightly run.