pytest Integration#

jamb integrates with pytest to link tests to requirements and generate traceability matrices.

Marking Tests#

Use @pytest.mark.requirement to link a test to one or more requirements:

import pytest

@pytest.mark.requirement("SRS001")
def test_single_requirement():
    assert True

@pytest.mark.requirement("SRS001", "SRS002")
def test_multiple_requirements():
    assert True

The marker accepts any number of item UIDs as positional arguments.

Supported Import Styles#

jamb recognizes the requirement marker in three import styles:

# Style 1: Fully qualified (recommended)
import pytest

@pytest.mark.requirement("SRS001")
def test_example(): ...

# Style 2: Import mark from pytest
from pytest import mark

@mark.requirement("SRS001")
def test_example(): ...

# Style 3: Import requirement directly
from pytest.mark import requirement

@requirement("SRS001")
def test_example(): ...

All three styles work with jamb check, pytest --jamb, and the automatic test file updates performed by jamb reorder.

Manual Test Case IDs#

By default, jamb assigns sequential test case IDs (TC001, TC002, etc.) automatically based on test order. To lock specific IDs to tests, use @pytest.mark.tc_id:

import pytest

@pytest.mark.tc_id("TC-AUTH-001")
@pytest.mark.requirement("SRS001")
def test_user_login():
    """Test with a manually assigned test case ID."""
    assert True

@pytest.mark.tc_id("TC042")
@pytest.mark.requirement("SRS002")
def test_password_reset():
    """Another test with a fixed ID."""
    assert True

Manual TC IDs:

Override auto-generated IDs for the marked test
Can use any format (TC001, TC-AUTH-001, TEST-42, etc.)
Reserve their number from auto-generation (e.g., @pytest.mark.tc_id("TC005") prevents auto-assignment of TC005)
Must be unique across all tests

To convert auto-generated IDs to permanent manual IDs, use jamb lock-tc after running tests. See the lock-tc command for details.

pytest Command-Line Options#

Option	Description
`--jamb`	Enable traceability checking
`--jamb-test-matrix PATH`	Output test records matrix to PATH (format inferred from extension)
`--jamb-trace-matrix PATH`	Output traceability matrix to PATH (format inferred from extension)
`--jamb-fail-uncovered`	Fail if any test spec items lack coverage
`--jamb-documents PREFIXES`	Comma-separated document prefixes to check
`--jamb-tester-id ID`	Tester identification for matrices (default: “Unknown”)
`--jamb-software-version VERSION`	Software version for matrices (overrides pyproject.toml)
`--trace-from PREFIX`	Starting document prefix for full chain trace matrix (e.g., UN, SYS)
`--include-ancestors`	Include “Traces To” column showing ancestors of starting items

Note: All pytest CLI options override their corresponding [tool.jamb] settings in pyproject.toml. For example, --jamb-fail-uncovered on the command line takes effect even if fail_uncovered = false in the config. When no CLI flag is given, the config file value is used.

Format inference: Matrix format is automatically inferred from the file extension: .html for HTML, .json for JSON, .csv for CSV, .md for Markdown, .xlsx for Excel.

Examples#

# Enable traceability checking
pytest --jamb

# Generate HTML trace matrix (format inferred from .html extension)
pytest --jamb --jamb-trace-matrix matrix.html

# Generate markdown trace matrix
pytest --jamb --jamb-trace-matrix matrix.md

# Generate test records matrix
pytest --jamb --jamb-test-matrix test-records.html

# Generate both matrices
pytest --jamb --jamb-trace-matrix trace.html --jamb-test-matrix records.html

# Fail if requirements lack coverage
pytest --jamb --jamb-fail-uncovered

# Check specific documents only
pytest --jamb --jamb-documents SRS

# Generate matrix with IEC 62304 metadata
pytest --jamb --jamb-trace-matrix matrix.html \
    --jamb-tester-id "CI Pipeline" \
    --jamb-software-version "1.2.3"

# Full chain trace from user needs with ancestor column
pytest --jamb --jamb-trace-matrix matrix.html \
    --trace-from UN \
    --include-ancestors

The `jamb_log` Fixture#

The jamb_log fixture provides structured test record logging aligned with IEC 62304 §5.7.5. Use it to document test actions, expected results, actual results, and notes that appear in the traceability matrix.

Methods#

Method	Matrix Column	Purpose
`jamb_log.test_action(text)`	Test Actions	What the test does (steps performed)
`jamb_log.expected_result(text)`	Expected Results	What should happen (acceptance criteria)
`jamb_log.actual_result(text)`	Actual Results	What actually happened (observed outcomes)
`jamb_log.note(text)`	Notes	Free-form observations or context

Example#

import pytest

@pytest.mark.requirement("SRS005")
def test_heart_rate_boundaries(jamb_log):
    """Test heart rate validation at boundary values."""
    jamb_log.test_action("Submit heart rate at lower boundary (30 BPM)")
    jamb_log.expected_result("Value is accepted as valid")
    result = validate_heart_rate(30)
    jamb_log.actual_result(f"validate_heart_rate(30) returned {result}")
    assert result is True

    jamb_log.test_action("Submit heart rate below minimum (29 BPM)")
    jamb_log.expected_result("Value is rejected as invalid")
    result = validate_heart_rate(29)
    jamb_log.actual_result(f"validate_heart_rate(29) returned {result}")
    assert result is False

    jamb_log.note("Boundary values selected per risk analysis RA-005")

Automatic Failure Capture#

When a test fails, jamb automatically captures the failure message and includes it in the Notes column of the matrix. No extra code needed.

Static vs Runtime Checking#

Feature	`jamb check` (static)	`pytest --jamb` (runtime)	`jamb matrix` (post-run)
Runs tests	No	Yes	No
Detects `@pytest.mark.requirement`	Yes	Yes	Uses saved data
Reports pass/fail status	No	Yes	Uses saved data
Captures `jamb_log` output	No	Yes	Uses saved data
Generates traceability matrix	No	Yes (with `--jamb-trace-matrix`)	Yes
Generates test records matrix	No	Yes (with `--jamb-test-matrix`)	Yes (with `--test-records`)
Speed	Fast	Depends on test suite	Fast

Use jamb check for quick feedback during development. Use pytest --jamb in CI for full traceability evidence. Use jamb matrix to regenerate matrices from saved coverage data without re-running tests.

Matrix Output Formats#

Matrix format is inferred from the file extension:

Format	Extension	Best For
HTML	`.html`	Regulatory submissions, standalone viewing
Markdown	`.md`	GitHub/GitLab rendering
JSON	`.json`	Tooling integration, programmatic access
CSV	`.csv`	Spreadsheet import, large datasets
XLSX	`.xlsx`	Excel, stakeholder review

Performance Note: When generating HTML or XLSX matrices with more than 5,000 rows, jamb emits a warning recommending CSV format for better performance and memory usage.

Format Details#

Matrix Metadata#

When using --jamb-tester-id and --jamb-software-version, the matrix includes a metadata header section with:

Software Version: The version being tested (from CLI flag, [tool.jamb].software_version, or [project].version)
Tester: Who ran the tests (from --jamb-tester-id)
Date: When the tests were executed (ISO 8601 UTC timestamp)
Environment: Test environment details (OS, Python version, platform, processor, hostname, CPU count)
Test Tools: Versions of all pytest plugins used (pytest, jamb, pytest-cov, etc.)

This metadata satisfies IEC 62304 Clause 5.7.5 requirements for software system test records.

Test Records Matrix (IEC 62304 5.7.5)#

The test records matrix is a test-centric view designed to satisfy IEC 62304 Clause 5.7.5 requirements for software system test records. Each row represents a single test execution with complete traceability to requirements.

IEC 62304 5.7.5 requires:

Test procedures or test cases
Expected results
Actual results
Pass/fail status
Tester identification
Date of testing

The jamb test records matrix provides all of these through:

Test Case column: Unique test identifier (TC001, etc.)
Expected Results column: Populated via jamb_log.expected_result()
Actual Results column: Populated via jamb_log.actual_result()
Outcome column: Pass/fail/skip status from pytest
Metadata header: Tester ID, timestamp, and software version
Requirements column: Links to verified requirements

Example output (one row per test):

Test Case	Test Name	Outcome	Requirements	Test Actions	Expected Results	Actual Results	Notes	Timestamp
TC001	test_credential_validation	passed	SRS001	Submit valid credentials	Authentication returns True	auth_user() returned True	Verified both auth paths	2024-01-15T10:30:00Z
TC002	test_account_lockout	failed	SRS002	Submit 3 invalid passwords	Account is locked	account.locked = False	[FAILURE] AssertionError	2024-01-15T10:30:05Z
TC003	test_heart_rate_boundaries	passed	SRS005	Submit heart rate at boundary	Value is accepted	validate_heart_rate(30) returned True	Boundary values per RA-005	2024-01-15T10:30:10Z

Trace Matrix Example#

The trace matrix is requirement-centric, showing the full traceability chain from a starting document through the hierarchy:

Traces To	UN	SYS	SRS	Tests	Status
-	UN001: User Authentication	SYS001: Authentication System	SRS001: The system shall authenticate users	TC001: test_credential_validation [passed]	Passed
-	UN001: User Authentication	SYS001: Authentication System	SRS002: The system shall lock accounts	TC002: test_account_lockout [failed]	Failed
-	UN002: Heart Rate Monitoring	SYS002: Vital Signs Display	SRS003: Display heart rate in real-time	-	Not Covered
-	UN003: Documentation	SYS003: User Manual	SRS099: User manual shall be provided	-	N/A

Status Values:

Passed (green): All linked tests passed
Failed (red): One or more linked tests failed
Partial (orange): Mix of passed and failed tests
Skipped (amber): All linked tests were skipped (none passed, none failed)
Not Covered (yellow): No tests linked to this requirement
N/A (gray): Requirement marked as testable: false (verified by other means)

HTML – Standalone document with inline CSS. Includes summary statistics banner, styled tables, and color-coded status. Test records matrix shows test actions, expected/actual results, and notes in separate columns. Trace matrix shows document hierarchy columns with bold UIDs and full requirement text.

Markdown – Pipe-delimited table with summary section. Test records show one row per test with semicolon-separated actions/results. Trace matrix shows document hierarchy with combined UID and text.

JSON – Structured data with arrays of test records or trace chain objects. Test records include test_actions, expected_results, actual_results, and notes arrays. Trace matrix includes nested chain objects per document level. Suitable for custom tooling.

CSV – Standard comma-separated values with summary rows at the top. Test records show one row per test. Trace matrix shows document hierarchy as separate columns. Recommended for large datasets (5,000+ rows).

XLSX – Excel workbook with summary rows, text wrapping, and color-coded status cells. Test records and trace matrices are formatted with appropriate column widths and styling.

Matrix Columns#

Test Records Matrix Columns#

Column	Description
Test Case	Sequential test ID (TC001, TC002, etc.) or manual ID from `@pytest.mark.tc_id`
Test Name	pytest function name
Outcome	Test result: passed, failed, skipped, or error
Requirements	Item UIDs covered by this test
Test Actions	Steps performed (from `jamb_log.test_action()`)
Expected Results	Acceptance criteria (from `jamb_log.expected_result()`)
Actual Results	Observed outcomes (from `jamb_log.actual_result()`)
Notes	Observations and failure messages (from `jamb_log.note()`)
Timestamp	ISO 8601 UTC timestamp of test execution

Parameterized Tests: When a test uses @pytest.mark.parametrize, each parameter variation receives a unique TC ID with an alphabetic suffix. For example, if test_boundary_values has three parameter sets, they become TC003a, TC003b, and TC003c. Suffixes follow the pattern: a-z, then aa, ab, etc. for tests with more than 26 parameters.

Trace Matrix Columns#

Column	Description
Traces To	Ancestor UIDs (optional, enabled with `--include-ancestors`)
[Document Columns]	One column per document in hierarchy (e.g., UN, SYS, SRS), showing UID and requirement text
Review Status	Always present. Shows “Reviewed”, “Not Reviewed”, or “Suspect” based on the item’s review state
[Extra Columns]	Custom attribute columns configured via `[[tool.jamb.matrix_columns]]` in pyproject.toml (see Configuration)
Tests	pytest tests linked to leaf items, with TC IDs and outcomes
Status	Passed, Failed, Not Covered, Partial, or N/A