Skip to content

Run your tests on your runner

Certyn runs your suite (Selenium Java or C#, k6 and others) from a runner inside your network, with tools and caches kept between runs.

For teams whose app is only on a VPN or internal network, or whose tests need tools that take long to install (a JDK, the .NET SDK, k6). A self-hosted runner puts Certyn's Machine on your own host.

Your setup

  • Tests in a repo: Selenium with Java and Maven, Selenium with C# and NUnit, a k6 script, or any other suite.
  • A Linux host with Docker that can reach the app. The runner needs only outbound HTTPS.
  • Your current CI (for example Jenkins) stays as it is.

Say this to Certyn

Our Selenium Java tests (Maven) run in Jenkins. The app is only on our VPN at https://staging.acme.internal. Run them and tell me why they fail.

Other examples:

  • Run our NUnit Selenium tests from github.com/acme/e2e on our runner against staging.
  • Run our k6 script loadtests/checkout.js on our runner against staging and tell me which thresholds failed.

What Certyn does

  1. If you have no runner yet, opens the runner form. It shows one docker run command. Before you run it on your host, add this line so tools and caches stay between runs:

      -e WORKSPACE_PATH=/srv/certyn-workspace \
    

    The form waits until the runner is online, then sends the default environment's runs to it.

  2. Opens a form for repo access: GH_TOKEN for GitHub, or your host's token (for example BITBUCKET_TOKEN) as a secret project variable.

  3. Starts a Machine on the runner and checks the app URL from inside your network.

  4. Installs what the suite needs into /workspace once (see the table). Package caches go there without extra setup: npm, Playwright browsers, pip, Maven (/workspace/.m2), Gradle and NuGet (/workspace/.nuget).

  5. Reads your CI file (Jenkinsfile, workflow) or build files for the test command, passes the environment URL ($TARGET_BASE_URL) to the setting your tests read, and runs one test.

  6. Runs the whole suite as a background run, and reads its report to decide the cause of each failure.

Run commands

FrameworkInstalled once into /workspaceTypical command
Selenium (Java, Maven)A JDK and Mavenmvn test (Certyn reads the Surefire XML)
Selenium (C#, NUnit)The .NET SDK version from global.json or the project filesdotnet test --logger "junit;LogFilePath=junit.xml" (or a TRX logger)
k6The k6 binaryk6 run -e BASE_URL=$TARGET_BASE_URL loadtests/checkout.js
Playwright, Cypress, pytest, othersWhat a cloud Machine has, plus your packagesYour CI's command

For C#, the URL goes in as a .runsettings parameter or an environment variable, whichever your tests read. Chromium and chromedriver are on the Machine.

For k6, k6 exits with an error when a threshold fails. Certyn reads the k6 summary and reports each threshold, passed or failed, with the measured value, plus failed checks and error rates.

What you get

  • A run report with failures grouped by cause: product bug, broken test, flaky test, setup problem.
  • One finding per real product bug, with evidence.
  • Fast later runs: the tools, packages and clone are already in /workspace.
  • Your CI unchanged. To keep its history in Certyn too, push its JUnit or Surefire XML: see Push JUnit results.
  • If you ask: the same run on a schedule or after each deploy.

Machine size for k6 and heavy suites

By default a runner gives each Machine 1 CPU core and 2.5 GB of memory. Raise it on the runner command, up to 16 each:

  -e CONTAINER_CPU_CORES=4 -e CONTAINER_MEMORY_GB=8 \

Certyn is not a load-testing service. The load comes from one Machine: one container, one host, one IP address. That is enough for a baseline or a small load test. For high or distributed load, use a load-testing service. Put load on staging, not production.

Limits

  • The runner form routes only the project's default environment. For another environment, set Execution target to Self-hosted runner on that environment in Settings → Environments.
  • Without WORKSPACE_PATH, every tool and dependency installs again at every run.
  • All runs on this runner share the workspace folder. Give each runner pool its own folder, and use it only for your team.
  • Machines run Linux. Tests that need Windows (Windows-only APIs or a Windows browser) do not run. Another browser must be installed on the runner first.
  • One shell command can run 30 minutes; longer jobs run in the background. A run has 60 minutes by default, up to 4 hours.
  • On Certyn's cloud these suites work too, but the JDK or SDK installs again at every run, which adds minutes, and you cannot change the Machine's size.