Skip to main content

Test Report Artifact

Overview​

CI produces and retains a single test report artifact so that audits and debugging can rely on one place for outcome, duration, and identity (e.g. scenario, backend).

What Is Produced​

  • JSON report: _ci/test-results.json (from pytest-json-report or extractor). Contains per-test outcome, duration, and (when parametrization is used) scenario_id and backend in test ids.
  • Summary: _ci/test-results-summary.json — totals, percentiles, slowest tests, quality gates.
  • HTML report (optional): reports/test_report.html — human-readable report with same data.

Task-level JSON is written under each task’s reports/ (e.g. tasks/data_ingestion_transformation/reports/test_report.json when using pytest --json-report). The root aggregate is produced by scripts/test_report/aggregate_all.py, which combines Task 1, 3, and 4 reports into reports/all_tests_summary.json and reports/ALL_TESTS_SUMMARY.md.

What Is Retained​

  • CI artifact: The GitHub Actions workflow uploads _ci/test-results.json and _ci/test-results-summary.json as the test-results artifact. Download it from the workflow run for the branch/commit you care about.
  • Local: After make test, the same files are written under _ci/ and (per task) under tasks/<task>/reports/.

Using the Artifact for Audits​

  • Pass/fail: Use totals in the summary (passed, failed, skipped, error).
  • Quality gates: Use quality_gates in the summary for pass/fail/warn and reasons.
  • ETL metrics: Per-test metrics (e.g. valid_rows, quarantined_rows) are injected by Task 1 conftest into the JSON report’s call.metrics for each test. Use these to prove pipeline behavior for a given run.

Using the Artifact for Debugging​

  • Which test failed: Each test has a stable nodeid; with parametrized scenario tests, the parametrize ids include scenario_id and backend (e.g. a1_pandas, a2_spark), so the JSON report and HTML report show which (scenario, backend) failed.
  • Duration: Per-test duration is in the report; slowest tests are in the summary’s slowest_tests.
  • Logs: Link from the report to the CI job logs for the failing test’s stdout/stderr.

Aggregate report and quality gates​

From repo root, make test (or the root test script) runs task tests and can produce per-task reports/test_report.json. The script scripts/test_report/aggregate_all.py combines Task 1, 3, and 4 reports into reports/all_tests_summary.json and reports/ALL_TESTS_SUMMARY.md. Quality gates (fail on failure, warn on skip/duration) are applied in build_summary.py using scripts/test_report/quality_gates.yaml. CI should run the aggregate and fail or warn based on the combined result.

References​

© 2026 Stephen Adei•CC BY 4.0