Skip to content

feat(reports): JUnit XML has no classname, no per-file suites and no captured output #1016

Description

@Chemaclass

Problem

The JUnit XML report (src/reports/junit.sh:16) is minimal enough that CI tools cannot use it well.

Reading generate_junit_xml:

  • One flat <testsuite name="bashunit"> wrapping every test in the run. Jenkins, GitLab, Azure DevOps, Buildkite and the dorny/test-reporter action all group results by suite; every bashunit run collapses into a single undifferentiated bucket.
  • No classname attribute on <testcase>. This is the attribute most consumers group and de-duplicate by; without it they show blank package names or fall back to the file path inconsistently.
  • No <system-out> / <system-err>. The test's own output is captured by the runner and shown on the console, but the XML carries only the failure message, so the CI failure view loses the context.
  • <failure message="Test failed"> is a constant string — the real message is in the element body, but tools that show only @message display "Test failed" for every failure.
  • No timestamp, hostname, errors breakdown, or <properties>.

Proposal

  • One <testsuite> per test file, nested in <testsuites>, each with its own tests / failures / skipped / time / timestamp / name (the file path).
  • classname on every <testcase>, derived from the test file path (e.g. tests/unit/assert/core_test.shtests.unit.assert.core_test), with name staying the human-readable test name.
  • <failure message="<first line of the real message>" type="AssertionFailed"> with the full message in the body.
  • <system-out> with the test's captured stdout when there is any, <system-err> for stderr.
  • Aggregate totals on the outer <testsuites> element.

Keep backwards compatibility in mind: state in the CHANGELOG that the XML shape changed.

Where to change

  • src/reports/junit.sh:16 generate_junit_xml.
  • src/reports/collect.sh:54 add_test — check whether the captured output is already retained; if not, this issue includes retaining it for the report path only (the console already renders it).

Acceptance criteria

  • Output validates against the JUnit XSD used by dorny/test-reporter (or a documented equivalent)
  • One <testsuite> per test file, with correct per-file counts and times
  • Every <testcase> has classname, name, file, time
  • <failure message=…> carries the real message, body carries the full text
  • <system-out> present when the test produced output, absent when it did not
  • Existing escaping still holds: ANSI stripped, control characters dropped, XML entities escaped (__xml_escape, src/reports/junit.sh:6)
  • Aggregate totals on <testsuites> match the summary counters
  • Correct under --parallel (regression guard for fix(reports): every report format is empty under --parallel #1004)
  • The macOS Bash 5.3 env LC_ALL=C awk workaround (7 test failures on Bash 5.3 macOS (nix-shell) #912) is preserved
  • Unit tests in tests/unit/reports/ asserting the XML structure

Repo checklist (agent)

  • TDD: RED → GREEN → REFACTOR. Write the failing test first.
  • Bash 3.0+ only: no printf -v, no += append, no declare -A, no [[ ]], no ${var,,}, no &>>, no ${arr[-1]}. Expanding a possibly-empty array under set -u needs ${arr[@]+"${arr[@]}"}.
  • A new CLI flag must be wired in all of these or a parity test fails:
  • Gates: make sa, make lint, ./bashunit tests/, ./bashunit --parallel tests/. Never run shfmt -w.
  • Docs: update docs/command-line.md. Editing docs/assertions.md invalidates the bashunit doc acceptance snapshot — regenerate it.
  • CHANGELOG.md: add one line under ## Unreleased.
  • Fixtures under tests/acceptance/fixtures/ must not end in *test.sh.
  • One issue = one PR.

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Type

No type

Projects

  • Status
    Done

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions