Test framework support matrix
Every cell below is executed. Each is an end-to-end bleep test run against a
generated project — resolve, compile, link, discover, run — asserted on the JUnit XML
that came back. A framework cannot appear here without a test running it, because the
matrix suites are generated from the same data this page is.
Every framework, and where it runs
| Framework | Version | Also tested | JVM / Java | JVM / Kotlin | JVM / Scala | Scala.js | Kotlin/JS | Scala Native | Kotlin/Native |
|---|---|---|---|---|---|---|---|---|---|
munit | 1.3.4 | 1.0.0, 0.7.29 | ✅ | ✅ | ✅ | ||||
scalatest | 3.2.19 | 3.2.15, 3.1.4 | ✅ | ✅ | ✅ | ||||
utest | 0.9.1 | 0.8.5, 0.7.11 | ✅ | ✅ | ✅ | ||||
scalacheck | 1.18.1 | 1.17.1, 1.15.4 | ✅ | ✅ | ✅ | ||||
specs2 | 4.20.9 | 4.19.2 | ✅ | ✅ | |||||
minitest | 2.9.6 | 2.8.2 | ✅ | ✅ | |||||
zio-test | 2.1.14 | 2.0.22 | ✅ | ||||||
weaver | 0.8.4 | 0.8.3 | ✅ | ✅ | |||||
hedgehog | 0.10.1 | 0.9.0 | ✅ | ✅ | |||||
junit5 | 5.14.4 | 5.10.1, 5.7.2 | ✅ | ||||||
junit4 | 4.13.2 | 4.12 | ✅ | ||||||
junit3 | 4.13.2 | — | ✅ | ||||||
kotest | 6.2.4 | 5.9.1 | ✅ | ||||||
kotlin.test | 2.4.10 | 2.2.21 | ✅ | ✅ | ✅ | ||||
testng | 7.10.2 | 7.5 | ✅ | ||||||
jqwik | 1.9.2 | — | ✅ | ||||||
cucumber | 7.20.1 | — | ✅ | ||||||
spek | 2.0.19 | — | ✅ |
A blank cell means the framework publishes nothing for that target, not that bleep
fails there. Version is what every build checks and what the recordings below show.
Also tested are covered by the same assertions when the version sweep is run
(bleep test bleep-tests --only-tag matrix), so a break in one of those could sit
unnoticed for longer. Anything not listed is untested rather than known-broken;
nothing here restricts what you may depend on.
What each run asserts
Per framework, per target, from the JUnit XML:
- every passing test is reported, under its own name
- the failing test is reported as a failure, and the throwing test as not passing
- the counts on the
<testsuite>element agree with the cases beneath it - a second suite in the same project is not run when one suite was selected
- the test's own stdout reaches
<system-out> - an uncaught exception's message reaches the report
- skipped tests are reported as skipped, where the framework can express one
- a third suite, whose tests all pass, is selected alongside the other two and comes back passing — a run in which every suite is red cannot check that
Two phrases in the tables below are worth pinning down, because they are not the same thing:
- nothing attached. The framework decided the test failed without ever throwing, so no exception exists. Nothing was lost in transit; there was never anything to carry.
- an empty stand-in. The framework built a throwaway exception when it reported the result, purely to fill the field. It has no message and its stack runs through the framework's own reporter, so it looks like an answer and is not.
In both cases the test is still correctly reported as failed and the reason is still captured — just not where you would look first, which is what the where to read the reason row is for.
A suite that cannot be constructed is a failure mode of its own: its constructor or initializer throws, so there is no test to attach the failure to. What you are told varies more between frameworks than anything else on this page. Every framework is run with exactly that — a suite that throws while being built, alongside a working one — and in every case the working suite still ran and reported normally.
Every framework in detail
One entry per framework. Open it for what bleep knows about that framework, then open a target inside it for a recording of a real run on that target — the actual terminal output, colours included.
Recordings are of the version checked on every build. The other versions listed are covered by the same assertions when the sweep is run; they are not recorded, because a second recording of the same output teaches nothing.
munit 1.3.4 — JVM / Scala, Scala.js, Scala Native
Scala, via org.scalameta::munit:1.3.4.
Recorded and checked on every build at 1.3.4. Also tested against 1.0.0 and 0.7.29 when the sweep is run.
| Can express a skipped test | yes |
| Failure carries a stack trace | yes |
on JVM / Scala
| A failed assertion reports | the real exception |
| An uncaught exception reports | the real exception |
| A suite that fails to construct | reported with the thrown exception |
on Scala.js
| A failed assertion reports | the real exception |
| An uncaught exception reports | the real exception |
| A suite that fails to construct | hangs until the idle timeout, then reports a timeout |
on Scala Native
| A failed assertion reports | the real exception |
| An uncaught exception reports | the real exception |
| A suite that fails to construct | reported with the thrown exception |
scalatest 3.2.19 — JVM / Scala, Scala.js, Scala Native
Scala, via org.scalatest::scalatest:3.2.19.
Recorded and checked on every build at 3.2.19. Also tested against 3.2.15 and 3.1.4 when the sweep is run.
| Can express a skipped test | yes |
| Failure carries a stack trace | yes |
on JVM / Scala
| A failed assertion reports | the real exception |
| An uncaught exception reports | the real exception |
| A suite that fails to construct | reported with the thrown exception |
on Scala.js
| Scala 3 | Scala 2.13, 2.12 | |
|---|---|---|
| A failed assertion reports | the real exception | an empty stand-in |
| An uncaught exception reports | the real exception | the real exception |
| A suite that fails to construct | reported with the thrown exception | reported with the thrown exception |
| Where to read the reason | the failing test | the suite's captured output, not the failing test |
The exception is genuinely ScalaTest's TestFailedException — it is not a stand-in — but under Scala 2.13 it reaches you carrying nothing you can act on: ScalaTest's assertion failures have no message, and the frame for your own test compiles to an anonymous JavaScript function (<jscode>.{anonymous}()), so the stack names only ScalaTest's internals. Under Scala 3 the same test keeps a frame naming your test class, which is why the two are separate columns above. The reason is in the suite's captured output — its <system-out> section, shown on the suite rather than on the failing test.
on Scala Native
| A failed assertion reports | an empty stand-in |
| An uncaught exception reports | an empty stand-in |
| A suite that fails to construct | reported, but without what was thrown |
| Where to read the reason | the suite's captured output, not the failing test |
ScalaTest's own Scala Native reporter is the cause, not the platform: SbtReporter.getOptionalThrowable hands over a freshly built, empty java.lang.Throwable instead of the exception that failed the test — the stack on it is that construction, identical whether the test failed an assertion or threw. The real exception never leaves ScalaTest, so there is nothing for bleep to recover; fixing it means fixing ScalaTest. The reason is in the suite's captured output — its <system-out> section, shown on the suite rather than on the failing test, which still names the exception type and the source line. For its full text, run the same test on the JVM or Scala.js.
utest 0.9.1 — JVM / Scala, Scala.js, Scala Native
Scala, via com.lihaoyi::utest:0.9.1.
Recorded and checked on every build at 0.9.1. Also tested against 0.8.5 and 0.7.11 when the sweep is run.
| Can express a skipped test | no — the framework has no such concept |
| Failure carries a stack trace | yes |
on JVM / Scala
| A failed assertion reports | the real exception |
| An uncaught exception reports | the real exception |
| A suite that fails to construct | reported with the thrown exception |
on Scala.js
| A failed assertion reports | the real exception |
| An uncaught exception reports | the real exception |
| A suite that fails to construct | reported with the thrown exception |
on Scala Native
| A failed assertion reports | the real exception |
| An uncaught exception reports | the real exception |
| A suite that fails to construct | reported with the thrown exception |
scalacheck 1.18.1 — JVM / Scala, Scala.js, Scala Native
Scala, via org.scalacheck::scalacheck:1.18.1.
Recorded and checked on every build at 1.18.1. Also tested against 1.17.1 and 1.15.4 when the sweep is run.
| Can express a skipped test | no — the framework has no such concept |
| Failure carries a stack trace | yes |
on JVM / Scala
| A failed assertion reports | the real exception |
| An uncaught exception reports | the real exception |
| A suite that fails to construct | reported, but without what was thrown |
on Scala.js
| A failed assertion reports | the real exception |
| An uncaught exception reports | the real exception |
| A suite that fails to construct | reported with the thrown exception |
on Scala Native
| A failed assertion reports | the real exception |
| An uncaught exception reports | the real exception |
| A suite that fails to construct | reported, but without what was thrown |
specs2 4.20.9 — JVM / Scala, Scala.js
Scala, via org.specs2::specs2-core:4.20.9.
Recorded and checked on every build at 4.20.9. Also tested against 4.19.2 when the sweep is run.
| Can express a skipped test | yes |
| Failure carries a stack trace | yes |
on JVM / Scala
| A failed assertion reports | the real exception |
| An uncaught exception reports | the real exception |
| A suite that fails to construct | reported with the thrown exception |
on Scala.js
| A failed assertion reports | the real exception |
| An uncaught exception reports | the real exception |
| A suite that fails to construct | reported with the thrown exception |
minitest 2.9.6 — JVM / Scala, Scala.js
Scala, via io.monix::minitest:2.9.6.
Recorded and checked on every build at 2.9.6. Also tested against 2.8.2 when the sweep is run.
| Can express a skipped test | no — the framework has no such concept |
| Failure carries a stack trace | no |
A warning about minitest. A suite that throws while being constructed is reported as a suite that
simply has no tests in it. Those two are indistinguishable from bleep's side, and a
class with no tests in it is a perfectly ordinary thing to have — so bleep cannot fail
the build on it without failing every unwritten test class too. The consequence is
worth knowing: a test class that blows up on construction leaves a green build. That
is a bug in the framework, not a policy choice by bleep.
on JVM / Scala
| A failed assertion reports | the real exception |
| An uncaught exception reports | the real exception |
| A suite that fails to construct | not reported at all — the framework swallows it |
on Scala.js
| A failed assertion reports | the real exception |
| An uncaught exception reports | the real exception |
| A suite that fails to construct | not reported at all — the framework swallows it |
zio-test 2.1.14 — JVM / Scala
Scala, via dev.zio::zio-test:2.1.14 and dev.zio::zio-test-sbt:2.1.14.
Recorded and checked on every build at 2.1.14. Also tested against 2.0.22 when the sweep is run.
| Can express a skipped test | yes |
| Failure carries a stack trace | yes |
on JVM / Scala
| A failed assertion reports | the real exception |
| An uncaught exception reports | the real exception |
| A suite that fails to construct | reported with the thrown exception |
weaver 0.8.4 — JVM / Scala, Scala.js
Scala, via com.disneystreaming::weaver-cats:0.8.4.
Recorded and checked on every build at 0.8.4. Also tested against 0.8.3 when the sweep is run.
| Can express a skipped test | no — the framework has no such concept |
| Failure carries a stack trace | yes |
on JVM / Scala
| A failed assertion reports | nothing attached |
| An uncaught exception reports | the real exception |
| A suite that fails to construct | reported with the thrown exception |
| Where to read the reason | the suite's captured error output, not the failing test |
A failed assertion here is an ordinary value rather than a thrown exception, so there is no exception in existence for bleep to attach — nothing is lost in transit, there was never anything to carry. The test is still correctly marked failed. The reason is in the suite's captured error output — its <system-err> section, which CI viewers usually show as a Standard error tab on the suite rather than on the failing test.
on Scala.js
| A failed assertion reports | nothing attached |
| An uncaught exception reports | the real exception |
| A suite that fails to construct | reported with the thrown exception |
| Where to read the reason | the suite's captured output, not the failing test |
A failed assertion here is an ordinary value rather than a thrown exception, so there is no exception in existence for bleep to attach — nothing is lost in transit, there was never anything to carry. The test is still correctly marked failed. The reason is in the suite's captured output — its <system-out> section, shown on the suite rather than on the failing test.
hedgehog 0.10.1 — JVM / Scala, Scala.js
Scala, via qa.hedgehog::hedgehog-sbt:0.10.1.
Recorded and checked on every build at 0.10.1. Also tested against 0.9.0 when the sweep is run.
| Can express a skipped test | no — the framework has no such concept |
| Failure carries a stack trace | yes |
on JVM / Scala
| A failed assertion reports | nothing attached |
| An uncaught exception reports | the real exception |
| A suite that fails to construct | reported with the thrown exception |
| Where to read the reason | the suite's captured output, not the failing test |
A failed assertion here is an ordinary value rather than a thrown exception, so there is no exception in existence for bleep to attach — nothing is lost in transit, there was never anything to carry. The test is still correctly marked failed. The reason is in the suite's captured output — its <system-out> section, shown on the suite rather than on the failing test.
on Scala.js
| A failed assertion reports | an empty stand-in |
| An uncaught exception reports | the real exception |
| A suite that fails to construct | reported with the thrown exception |
| Where to read the reason | the suite's captured output, not the failing test |
hedgehog's Scala.js adapter builds a MessageOnlyException in Event$.fromReport to satisfy the interface. It carries no message and its stack is that construction, so it looks like a report and tells you nothing. The reason is in the suite's captured output — its <system-out> section, shown on the suite rather than on the failing test.
junit5 5.14.4 — JVM / Java
Java, via org.junit.jupiter:junit-jupiter:5.14.4.
Recorded and checked on every build at 5.14.4. Also tested against 5.10.1 and 5.7.2 when the sweep is run.
| Can express a skipped test | yes |
| Failure carries a stack trace | yes |
on JVM / Java
| A failed assertion reports | the real exception |
| An uncaught exception reports | the real exception |
| A suite that fails to construct | reported with the thrown exception |
junit4 4.13.2 — JVM / Java
Java, via junit:junit:4.13.2.
Recorded and checked on every build at 4.13.2. Also tested against 4.12 when the sweep is run.
| Can express a skipped test | yes |
| Failure carries a stack trace | yes |
on JVM / Java
| A failed assertion reports | the real exception |
| An uncaught exception reports | the real exception |
| A suite that fails to construct | reported with the thrown exception |
junit3 4.13.2 — JVM / Java
Java, via junit:junit:4.13.2.
Only 4.13.2 is tested.
| Can express a skipped test | no — the framework has no such concept |
| Failure carries a stack trace | yes |
on JVM / Java
| A failed assertion reports | the real exception |
| An uncaught exception reports | the real exception |
| A suite that fails to construct | reported with the thrown exception |
kotest 6.2.4 — JVM / Kotlin
Kotlin, via io.kotest:kotest-runner-junit5-jvm:6.2.4.
Recorded and checked on every build at 6.2.4. Also tested against 5.9.1 when the sweep is run.
| Can express a skipped test | yes |
| Failure carries a stack trace | yes |
on JVM / Kotlin
| A failed assertion reports | the real exception |
| An uncaught exception reports | the real exception |
| A suite that fails to construct | reported with the thrown exception |
kotlin.test 2.4.10 — JVM / Kotlin, Kotlin/JS, Kotlin/Native
Kotlin, via org.jetbrains.kotlin:kotlin-test-junit5:2.4.10.
Recorded and checked on every build at 2.4.10. Also tested against 2.2.21 when the sweep is run.
| Can express a skipped test | yes |
| Failure carries a stack trace | yes |
on JVM / Kotlin
| A failed assertion reports | the real exception |
| An uncaught exception reports | the real exception |
| A suite that fails to construct | reported with the thrown exception |
on Kotlin/JS
| A failed assertion reports | the real exception |
| An uncaught exception reports | the real exception |
| A suite that fails to construct | reported with the thrown exception |
on Kotlin/Native
| A failed assertion reports | the real exception |
| An uncaught exception reports | the real exception |
| A suite that fails to construct | reported with the thrown exception |
testng 7.10.2 — JVM / Java
Java, via org.testng:testng:7.10.2 and com.lihaoyi:mill-contrib-testng_2.13:0.9.6.
Recorded and checked on every build at 7.10.2. Also tested against 7.5 when the sweep is run.
| Can express a skipped test | yes |
| Failure carries a stack trace | yes |
on JVM / Java
| A failed assertion reports | the real exception |
| An uncaught exception reports | the real exception |
| A suite that fails to construct | reported, but without what was thrown |
jqwik 1.9.2 — JVM / Java
Java, via net.jqwik:jqwik:1.9.2.
Only 1.9.2 is tested.
| Can express a skipped test | yes |
| Failure carries a stack trace | yes |
on JVM / Java
| A failed assertion reports | the real exception |
| An uncaught exception reports | the real exception |
| A suite that fails to construct | reported with the thrown exception |
cucumber 7.20.1 — JVM / Java
Java, via io.cucumber:cucumber-java:7.20.1 and io.cucumber:cucumber-junit-platform-engine:7.20.1 and org.junit.platform:junit-platform-suite:1.11.4.
Only 7.20.1 is tested.
| Can express a skipped test | no — the framework has no such concept |
| Failure carries a stack trace | yes |
on JVM / Java
| A failed assertion reports | the real exception |
| An uncaught exception reports | the real exception |
| A suite that fails to construct | not applicable — the discovered class is a suite declaration, not a test class |
spek 2.0.19 — JVM / Kotlin
Kotlin, via org.spekframework.spek2:spek-dsl-jvm:2.0.19 and org.spekframework.spek2:spek-runner-junit5:2.0.19 and org.jetbrains.kotlin:kotlin-test:2.4.10.
Only 2.0.19 is tested.
| Can express a skipped test | yes |
| Failure carries a stack trace | yes |
on JVM / Kotlin
| A failed assertion reports | the real exception |
| An uncaught exception reports | the real exception |
| A suite that fails to construct | reported with the thrown exception |