Inside Test Frameworks — Designing and Building a Test Runner from Scratch

When you type pytest, the test files are found, the test functions run in order, and a failed comparison is reported along with the values it compared.

Test frameworks (pytest, unittest, Jest, Go's testing package, etc.) automatically discover, organize, execute, and report on tests. In this article, we build a small test runner in Python that performs each of these steps and look at how each one is designed.

The Big Picture of a Test Framework

A test framework's work can be divided into the following pipeline.

The interactive demo below lets you step through what the test runner does at each phase.

Test Runner Internals

Follows two test files from discovery to the report, one step at a time

Test DiscoveryScanning for test files using glob patterns...

Test Registry

(empty)

Internal Log

> rglob("test_*.py")

📁 Found test_math.py

📁 Found test_string.py

1/10 — Test Discovery1 / 10

Phase 1: Test Discovery — Finding the Files

A test framework first has to find which files contain tests.

Convention-Based Discovery

Most frameworks identify test files by filename conventions.

pytest:
  test_*.py
  *_test.py
 
unittest:
  test*.py (default for discover)
 
Jest / Vitest:
  **/*.test.{ts,js}
  **/*.spec.{ts,js}
 
Go:
  *_test.go

Our implementation follows pytest's conventions. pathlib.Path.rglob() traverses directories recursively and returns every file that matches a pattern.

# mini_test/discovery.py
from pathlib import Path
from fnmatch import fnmatch
 
DEFAULT_PATTERNS = ["test_*.py", "*_test.py"]
DEFAULT_IGNORE = ["__pycache__", ".venv", "venv", ".git"]
 
 
def discover_test_files(
    root_dir: str = ".",
    patterns: list[str] | None = None,
    ignore: list[str] | None = None,
) -> list[Path]:
    """Recursively find test files matching glob patterns"""
    patterns = patterns or DEFAULT_PATTERNS
    ignore = ignore or DEFAULT_IGNORE
    root = Path(root_dir)
 
    files: list[Path] = []
    for path in root.rglob("*.py"):
        # Skip directories that should be ignored
        if any(part in ignore for part in path.parts):
            continue
        # Check if filename matches any pattern
        if any(fnmatch(path.name, pat) for pat in patterns):
            files.append(path)
 
    return sorted(files)

We apply sorted() to the result to make the test execution order deterministic. The order in which the filesystem returns files varies by OS and filesystem type; sorting gives the same order in every environment. The ignore list (__pycache__, .venv, etc.) keeps the runner from picking up .py files inside those directories as tests.

Why Glob Patterns?

Glob patterns are declarative. Users write the rules for where test files live in a configuration file, and the framework only has to scan the filesystem with those patterns. In pytest, the following settings play this role.

# pyproject.toml — testpaths sets where to search; python_* set the name patterns
[tool.pytest.ini_options]
testpaths = ["tests"]
python_files = ["test_*.py", "*_test.py"]
python_classes = ["Test*"]
python_functions = ["test_*"]

Phase 2: Test Collection — Gathering Test Functions from Modules

Once test files are found, the next step is to collect the test cases inside them.

pytest's Collection Strategy: Function Name Conventions

pytest doesn't use DSLs like describe/it. Standard function tests are identified by function name conventions (test_ prefix). Real pytest also supports Test* classes, unittest.TestCase, and configuration overrides such as python_files, python_classes, and python_functions. Because names alone identify tests, a runner can load a module, list its functions with Python's inspect module, and keep the ones whose names match.

The implementation below uses importlib.util to import each test file dynamically as a module. It takes three steps: spec_from_file_location() creates a module spec from a file path, module_from_spec() builds the module object, and exec_module() executes the code. This lets us import any .py file without modifying sys.path.

# mini_test/collector.py
import importlib.util
import inspect
from dataclasses import dataclass, field
from typing import Callable
from pathlib import Path
 
 
@dataclass
class TestCase:
    name: str
    fn: Callable[[], None]
    module_name: str
 
 
@dataclass
class TestSuite:
    name: str
    tests: list[TestCase] = field(default_factory=list)
    setup_module: Callable[[], None] | None = None
    teardown_module: Callable[[], None] | None = None
    setup_function: Callable[[], None] | None = None
    teardown_function: Callable[[], None] | None = None
 
 
def collect_from_file(path: Path) -> TestSuite:
    """Load a test file and collect test_ functions"""
    # Dynamically import the file as a module
    spec = importlib.util.spec_from_file_location(path.stem, path)
    if spec is None or spec.loader is None:
        raise ImportError(f"Cannot load {path}")
 
    module = importlib.util.module_from_spec(spec)
    spec.loader.exec_module(module)  # ← top-level code executes here
 
    suite = TestSuite(name=path.stem)
 
    # Collect lifecycle hooks
    suite.setup_module = getattr(module, "setup_module", None)
    suite.teardown_module = getattr(module, "teardown_module", None)
    suite.setup_function = getattr(module, "setup_function", None)
    suite.teardown_function = getattr(module, "teardown_function", None)
 
    # Collect functions starting with test_
    for name, obj in inspect.getmembers(module, inspect.isfunction):
        if name.startswith("test_"):
            suite.tests.append(
                TestCase(name=name, fn=obj, module_name=path.stem)
            )
 
    return suite

Separating Collection from Execution

Calling exec_module() executes the module's top-level code. The test functions themselves do not run at this point; inspect.getmembers() only collects them by name.

# test_math.py — this file gets exec_module()'d
 
import math
 
print("This line runs during the collection phase")  # module level
 
def test_addition():
    # ← collected only; not executed yet
    assert 1 + 1 == 2
 
def test_sqrt():
    # ← collected only; not executed yet
    assert math.sqrt(4) == 2.0
 
def helper_function():
    # ← not collected (doesn't start with test_)
    pass

The runner imports the module to collect the tests and runs the test functions later. This separation is the foundation of pytest-style frameworks. It allows a runner to:

Know the total number of tests before running them (for progress bars)
Filter (-k option) and reorder tests before running them
Insert lifecycle hooks (setup/teardown) at the right moments

Lifecycle Hook Registration

pytest recognizes module-level functions with fixed names, such as setup_module, as xunit-style lifecycle hooks (unittest uses names such as setUpModule and TestCase.setUp). Because the hooks are split into module-level setup_module/teardown_module and per-test setup_function/teardown_function, each operation can run at the frequency it needs. Expensive resources such as database connections are created and destroyed once per module, while operations that keep tests independent, such as clearing tables, run before every test.

# test_database.py — lifecycle hook example
 
db = None
 
def setup_module():
    """Runs once before all tests in the module"""
    global db
    db = create_test_database()
 
def teardown_module():
    """Runs once after all tests in the module"""
    db.drop()
 
def setup_function():
    """Runs before each test function"""
    db.clear_tables()
 
def teardown_function():
    """Runs after each test function"""
    db.rollback()
 
def test_insert():
    db.insert({"key": "value"})
    assert db.get("key") == "value"
 
def test_delete():
    db.insert({"key": "value"})
    db.delete("key")
    assert db.get("key") is None

Phase 3: The Execution Engine — Running in the Right Order

After collection, the runner executes the tests. It runs each test in isolation from the others and calls the lifecycle hooks around it in the correct order.

Execution Order

A suite runs in a nested order: setup_module → (setup_function → test body → teardown_function) × n → teardown_module.

Execution flow for Suite "test_math":
  ┌─ setup_module()         ← once for the entire module
  │
  │  ┌─ setup_function()    ← once per test
  │  │  test_addition() runs
  │  └─ teardown_function()
  │
  │  ┌─ setup_function()
  │  │  test_sqrt() runs
  │  └─ teardown_function()
  │
  └─ teardown_module()      ← once for the entire module

Runner Implementation

The runner measures durations with time.perf_counter(). time.time() follows the system clock, so NTP synchronization or manual clock changes can affect it. perf_counter() uses a monotonic clock, which makes it suitable for measuring elapsed time.

# mini_test/runner.py
import time
from dataclasses import dataclass
 
 
@dataclass
class TestResult:
    name: str
    suite_name: str
    status: str  # "passed" or "failed"
    duration: float
    error: Exception | None = None
 
 
def run_suite(suite: TestSuite) -> list[TestResult]:
    results: list[TestResult] = []
 
    # setup_module — module-wide initialization
    if suite.setup_module:
        suite.setup_module()
 
    for test_case in suite.tests:
        # setup_function — per-test initialization
        if suite.setup_function:
            suite.setup_function()
 
        start = time.perf_counter()
        try:
            # Execute the test body
            test_case.fn()
            results.append(TestResult(
                name=test_case.name,
                suite_name=suite.name,
                status="passed",
                duration=time.perf_counter() - start,
            ))
        except Exception as exc:
            results.append(TestResult(
                name=test_case.name,
                suite_name=suite.name,
                status="failed",
                duration=time.perf_counter() - start,
                error=exc,
            ))
 
        # teardown_function — per-test cleanup
        if suite.teardown_function:
            suite.teardown_function()
 
    # teardown_module — module-wide cleanup
    if suite.teardown_module:
        suite.teardown_module()
 
    return results

Test Isolation via try-except

A basic design principle of a test runner is that one test's failure must not spread to other tests.

# Each test is wrapped in its own try-except
try:
    test_case.fn()
    # → if no exception is thrown, PASS
except Exception as exc:
    # → if an exception is thrown, FAIL (other tests unaffected)

This works because assertions do nothing more than raise exceptions. assert 1 == 2 internally executes raise AssertionError. A test passes if it finishes without raising and fails if an exception is raised.

except Exception catches assertion failures and also unexpected ordinary exceptions such as TypeError and KeyError. When a test raises one of these, the runner records the test as failed and continues with the remaining tests. KeyboardInterrupt and SystemExit do not inherit from Exception, so this except clause does not catch them, and the whole run stops. pytest and unittest stop the run on a KeyboardInterrupt from Ctrl+C, but treat a SystemExit raised inside a test like any other exception from that test.

Phase 4: The Assertion Engine — Inside assert

The assertion engine is the part of a test framework that test authors use directly.

Python's assert Statement

In Python tests, the built-in assert statement works out of the box. It is syntactic sugar for raising AssertionError when its condition is false.

# The essence of assert
assert 1 + 1 == 2
# ↓ equivalent to
if not (1 + 1 == 2):
    raise AssertionError

However, a plain assert reports almost nothing when it fails. For example, when assert result == "Hello, world!" fails, plain Python shows only AssertionError, with no indication of what result contained. pytest solves this through assert rewriting.

pytest's Assert Rewriting

pytest rewrites the AST (abstract syntax tree) of assert statements at import time so that a failure reports details such as the compared values.

# What the user writes
def test_greeting():
    result = greet("world")
    assert result == "Hello, world!"
 
# What pytest internally transforms it into (conceptually)
def test_greeting():
    result = greet("world")
    _left = result
    _right = "Hello, world!"
    if not (_left == _right):
        raise AssertionError(
            f"assert {_left!r} == {_right!r}\n"
            f"  where {_left!r} = greet('world')"
        )

Here is a simplified implementation. It handles only assert statements with a single comparison operator. Each one is replaced with code that stores the left and right operands in temporary variables and, if the comparison is false, raises an AssertionError whose message includes both values.

# mini_test/assertion_rewrite.py
import ast
import textwrap
 
 
class AssertRewriter(ast.NodeTransformer):
    """Transform assert statements to include detailed error messages"""
 
    def visit_Assert(self, node: ast.Assert) -> ast.AST:
        # Only rewrite assert a == b patterns
        if isinstance(node.test, ast.Compare):
            return self._rewrite_compare(node)
        return node
 
    def _rewrite_compare(self, node: ast.Assert) -> list[ast.stmt]:
        compare = node.test
        if not isinstance(compare, ast.Compare) or len(compare.ops) != 1:
            return [node]
 
        # Save left and right values in temporary variables
        stmts: list[ast.stmt] = []
 
        # __mt_left = <left expr>
        stmts.append(ast.Assign(
            targets=[ast.Name(id="__mt_left", ctx=ast.Store())],
            value=compare.left,
            lineno=node.lineno,
            col_offset=node.col_offset,
        ))
 
        # __mt_right = <right expr>
        stmts.append(ast.Assign(
            targets=[ast.Name(id="__mt_right", ctx=ast.Store())],
            value=compare.comparators[0],
            lineno=node.lineno,
            col_offset=node.col_offset,
        ))
 
        # if not (__mt_left <op> __mt_right): raise AssertionError(...)
        op_symbols = {
            "Eq": "==", "NotEq": "!=", "Lt": "<", "LtE": "<=",
            "Gt": ">", "GtE": ">=", "Is": "is", "IsNot": "is not",
            "In": "in", "NotIn": "not in",
        }
        op_name = type(compare.ops[0]).__name__
        op_symbol = op_symbols.get(op_name, op_name)
        raise_node = ast.Raise(
            exc=ast.Call(
                func=ast.Name(id="AssertionError", ctx=ast.Load()),
                args=[ast.JoinedStr(values=[
                    ast.Constant(value="assert "),
                    ast.FormattedValue(
                        value=ast.Name(id="__mt_left", ctx=ast.Load()),
                        conversion=ord("r"),
                    ),
                    ast.Constant(value=f" {op_symbol} "),
                    ast.FormattedValue(
                        value=ast.Name(id="__mt_right", ctx=ast.Load()),
                        conversion=ord("r"),
                    ),
                ])],
                keywords=[],
            ),
        )
        stmts.append(ast.If(
            test=ast.UnaryOp(
                op=ast.Not(),
                operand=ast.Compare(
                    left=ast.Name(id="__mt_left", ctx=ast.Load()),
                    ops=compare.ops,
                    comparators=[ast.Name(id="__mt_right", ctx=ast.Load())],
                ),
            ),
            body=[raise_node],
            orelse=[],
        ))
 
        # Assign line numbers to all nodes
        for stmt in stmts:
            ast.fix_missing_locations(stmt)
        return stmts

Custom Assert Functions

Detailed messages are also possible without rewriting the AST, by using custom assert functions.

With AST rewriting, users write a plain assert and still get detailed messages, but the implementation is complex and harder to debug. Custom assert functions require calls like assert_equal(a, b), but their implementation is simple and easy to follow.

# mini_test/assertions.py
from typing import Any
 
 
class AssertionError(Exception):
    def __init__(self, message: str, actual: Any = None, expected: Any = None):
        super().__init__(message)
        self.actual = actual
        self.expected = expected
 
 
def assert_equal(actual: Any, expected: Any) -> None:
    """Equality check using =="""
    if actual != expected:
        raise AssertionError(
            f"Expected {expected!r}, got {actual!r}",
            actual=actual,
            expected=expected,
        )
 
 
def assert_is(actual: Any, expected: Any) -> None:
    """Identity check using is"""
    if actual is not expected:
        raise AssertionError(
            f"Expected {expected!r} (id={id(expected)}), "
            f"got {actual!r} (id={id(actual)})",
            actual=actual,
            expected=expected,
        )
 
 
def assert_true(value: Any) -> None:
    if not value:
        raise AssertionError(f"Expected truthy, got {value!r}")
 
 
def assert_raises(exc_type: type, fn: Any, *args: Any, **kwargs: Any) -> None:
    """Verify that a function raises a specific exception"""
    try:
        fn(*args, **kwargs)
    except exc_type:
        return  # Expected exception → PASS
    except Exception as e:
        raise AssertionError(
            f"Expected {exc_type.__name__}, got {type(e).__name__}: {e}"
        )
    raise AssertionError(
        f"Expected {exc_type.__name__} to be raised, but nothing was raised"
    )
 
 
def assert_in(item: Any, container: Any) -> None:
    if item not in container:
        raise AssertionError(
            f"Expected {item!r} to be in {container!r}"
        )
 
 
def assert_length(obj: Any, length: int) -> None:
    actual = len(obj)
    if actual != length:
        raise AssertionError(
            f"Expected length {length}, got {actual}"
        )

== vs is — What's the Difference?

The difference between == and is often confuses test writers. Use == to check that values are equal, and is to check identity with a singleton such as None or a reference to the same object.

# == → value equality (__eq__ method)
assert [1, 2] == [1, 2]     # ✅ same values → OK
assert {"a": 1} == {"a": 1} # ✅ same structure → OK
 
# is → object identity (same id)
a = [1, 2]
b = [1, 2]
assert a is a  # ✅ same object
assert a is b  # ❌ same values but different objects
 
# Caveat: CPython caches small integers
assert 1 is 1        # ✅ CPython implementation detail (-5 to 256 are cached)
assert 1000 is 1000  # ⚠️ may happen to be True in CPython

assert with not

not has lower precedence than comparison operators such as ==, so assert not 1 == 2 evaluates not (1 == 2).

assert 1 == 2       → 1 == 2 is False → AssertionError
assert not 1 == 2   → not (1 == 2) is True → PASS
assert not 1 == 1   → not (1 == 1) is False → AssertionError

Phase 5: Mocking & Patching — Replacing Function Behavior

Test frameworks also provide ways to replace a function's behavior and record its calls.

Tests often run into external dependencies such as API calls, database connections, and filesystem operations. Including these dependencies directly makes tests slow, flaky, and environment-dependent. Mocks replace the dependencies with fakes so that a test checks only the logic under test.

Implementing a Spy

A spy wraps the original function and records the arguments and return value of each call. The Spy below can also replace the return value or the implementation, so it doubles as a mock.

# mini_test/spy.py
from typing import Any, Callable
from dataclasses import dataclass, field
 
_UNSET = object()  # Sentinel for "not set"
 
 
@dataclass
class CallRecord:
    args: tuple
    kwargs: dict
    return_value: Any = None
    exception: Exception | None = None
 
 
class Spy:
    """Wraps a function and records all calls"""
 
    def __init__(self, original: Callable | None = None):
        self._original = original
        self._mock_return: Any = _UNSET
        self._mock_impl: Callable | None = None
        self.calls: list[CallRecord] = []
 
    def __call__(self, *args: Any, **kwargs: Any) -> Any:
        record = CallRecord(args=args, kwargs=kwargs)
        self.calls.append(record)
 
        # If a mock return value was set
        if self._mock_return is not _UNSET:
            record.return_value = self._mock_return
            return self._mock_return
 
        # If a replacement implementation was set
        impl = self._mock_impl or self._original
        if impl:
            try:
                result = impl(*args, **kwargs)
                record.return_value = result
                return result
            except Exception as exc:
                record.exception = exc
                raise
 
        return None
 
    @property
    def call_count(self) -> int:
        return len(self.calls)
 
    def mock_return_value(self, value: Any) -> "Spy":
        self._mock_return = value
        return self
 
    def mock_implementation(self, fn: Callable) -> "Spy":
        self._mock_impl = fn
        self._mock_return = _UNSET
        return self
 
    def reset(self) -> None:
        self._mock_impl = None
        self._mock_return = _UNSET
        self.calls.clear()

Testing with Spies

def test_spy_records_calls():
    add = Spy(lambda a, b: a + b)
 
    add(1, 2)
    add(3, 4)
 
    assert add.call_count == 2
    assert add.calls[0].args == (1, 2)
    assert add.calls[0].return_value == 3
 
def test_spy_mock_return():
    fetch_user = Spy()
    fetch_user.mock_return_value({"name": "Alice"})
 
    user = fetch_user()
    assert user["name"] == "Alice"

Patching Object Attributes

In Python, the widely used unittest.mock.patch works by replacing attributes. The Patch class below is implemented as a context manager for use in a with statement. __enter__ replaces the attribute and __exit__ restores it, so the original is always restored when the with block exits. If restoration were forgotten, the replaced attribute would remain for later tests and break test isolation.

# mini_test/patch.py
from typing import Any
 
 
class Patch:
    """Context manager that temporarily replaces an object's attribute"""
 
    def __init__(self, obj: Any, attr: str, replacement: Any = None):
        self.obj = obj
        self.attr = attr
        self.replacement = replacement or Spy()
        self._original: Any = None
 
    def __enter__(self) -> Any:
        # Save the original attribute and replace it
        self._original = getattr(self.obj, self.attr)
        setattr(self.obj, self.attr, self.replacement)
        return self.replacement
 
    def __exit__(self, *exc_info: Any) -> None:
        # Restore the original attribute
        setattr(self.obj, self.attr, self._original)

# Usage example
import os
 
def test_patch_environ():
    with Patch(os, "getcwd", Spy(lambda: "/fake/path")) as mock_cwd:
        result = os.getcwd()
        assert result == "/fake/path"
        assert mock_cwd.call_count == 1
 
    # After exiting the with block, the original is automatically restored
    assert os.getcwd() != "/fake/path"

Phase 6: The Reporter — Communicating Results to Humans

The reporter formats test results as human-readable output.

Reporter Interface

Defining the reporter as an abstract class (interface) is a typical application of the Strategy pattern. The same results can be written to the console, a JSON file, JUnit XML, HTML, and other formats. The runner knows nothing about specific reporter implementations and depends only on the interface.

# mini_test/reporter.py
from abc import ABC, abstractmethod
 
 
class Reporter(ABC):
    @abstractmethod
    def on_suite_start(self, suite: TestSuite) -> None: ...
 
    @abstractmethod
    def on_test_result(self, result: TestResult) -> None: ...
 
    @abstractmethod
    def on_suite_end(self, suite: TestSuite, results: list[TestResult]) -> None: ...
 
    @abstractmethod
    def on_run_end(self, all_results: list[TestResult]) -> None: ...

Console Reporter Implementation

The console reporter uses ANSI escape codes (\033[32m for green, \033[31m for red, \033[0m for reset) to color-code test results. Passes appear as a green ✓ and failures as a red ✗, so failures are easy to spot even among many results.

# mini_test/console_reporter.py
import time
 
 
class ConsoleReporter(Reporter):
    def __init__(self):
        self._start_time = 0.0
 
    def on_suite_start(self, suite: TestSuite) -> None:
        print(f"\n  {suite.name}")
        self._start_time = time.perf_counter()
 
    def on_test_result(self, result: TestResult) -> None:
        icon = "✓" if result.status == "passed" else "✗"
        color = "\033[32m" if result.status == "passed" else "\033[31m"
        reset = "\033[0m"
        duration = f"{result.duration * 1000:.1f}"
 
        print(f"    {color}{icon}{reset} {result.name} ({duration}ms)")
 
        if result.error:
            print(f"      {result.error}")
 
    def on_suite_end(self, suite: TestSuite, results: list[TestResult]) -> None:
        pass  # Per-suite summary omitted for brevity
 
    def on_run_end(self, all_results: list[TestResult]) -> None:
        passed = sum(1 for r in all_results if r.status == "passed")
        failed = sum(1 for r in all_results if r.status == "failed")
        total = len(all_results)
        duration = (time.perf_counter() - self._start_time) * 1000
 
        print("\n" + "─" * 40)
        if failed > 0:
            print(f"\033[31m  Tests: {passed} passed, {failed} failed, {total} total\033[0m")
        else:
            print(f"\033[32m  Tests: {passed} passed, {total} total\033[0m")
        print(f"  Time:  {duration:.0f}ms")

Integrating the Reporter into the Runner

# mini_test/runner.py (with reporter integration)
 
def run(suites: list[TestSuite], reporter: Reporter) -> list[TestResult]:
    all_results: list[TestResult] = []
 
    for suite in suites:
        reporter.on_suite_start(suite)
        results = run_suite(suite)
 
        for result in results:
            reporter.on_test_result(result)
 
        reporter.on_suite_end(suite, results)
        all_results.extend(results)
 
    reporter.on_run_end(all_results)
    return all_results

Phase 7: Putting It All Together — The CLI Entry Point

Finally, we build a CLI entry point that ties all the phases together. It reports success or failure through its exit code. Exiting with sys.exit(1 if failed else 0) lets CI servers (GitHub Actions, Jenkins, CircleCI, etc.) determine test success from the exit code alone. This is the Unix convention of "exit code 0 = success, non-zero = failure", and pytest, Jest, and go test also exit with a non-zero code when tests fail.

# mini_test/cli.py
import sys
from mini_test.discovery import discover_test_files
from mini_test.collector import collect_from_file
from mini_test.runner import run
from mini_test.console_reporter import ConsoleReporter
 
 
def main() -> None:
    # Phase 1: Discovery
    files = discover_test_files()
    print(f"Found {len(files)} test file(s)\n")
 
    # Phase 2: Collection
    suites = [collect_from_file(f) for f in files]
 
    # Phase 3–6: Execution + Reporting
    reporter = ConsoleReporter()
    results = run(suites, reporter)
 
    # Exit code
    failed = any(r.status == "failed" for r in results)
    sys.exit(1 if failed else 0)
 
 
if __name__ == "__main__":
    main()

Sample Output

$ python -m mini_test
 
Found 2 test file(s)
 
  test_math
    ✓ test_addition (0.3ms)
    ✓ test_division_by_zero (0.1ms)
 
  test_string
    ✓ test_trim (0.2ms)
    ✗ test_upper (0.4ms)
      AssertionError: Expected 'HELLO', got 'Hello'
 
────────────────────────────────────────
  Tests: 3 passed, 1 failed, 4 total
  Time:  12ms

Features Real Frameworks Add

We have now implemented the core features of a test framework. On top of this core, real frameworks add parallelization to shorten run time, snapshot testing to detect output regressions, coverage measurement to find untested code, and watch mode to rerun tests when files change.

Test Parallelization

pytest-xdist uses execnet to spawn and manage worker processes and distributes tests across the workers to run them in parallel. For simplicity, the diagram and implementation below distribute work per test file.

Parallel execution architecture:
  Main process (coordinator)
    │
    ├── Worker 1 (subprocess): test_math.py
    ├── Worker 2 (subprocess): test_string.py
    ├── Worker 3 (subprocess): test_api.py
    └── Worker 4 (subprocess): test_ui.py
 
  Each worker is an independent process,
  preventing global state interference.
  Results are aggregated via IPC to the main process.

# Conceptual implementation of parallel execution
from multiprocessing import Pool
from pathlib import Path
 
def run_file(path: str) -> list[TestResult]:
    suite = collect_from_file(Path(path))
    return run_suite(suite)
 
def run_in_parallel(files: list[Path], num_workers: int = 4) -> list[TestResult]:
    with Pool(num_workers) as pool:
        nested = pool.map(run_file, [str(f) for f in files])
    return [r for results in nested for r in results]  # flatten

Snapshot Testing

Snapshot testing saves the output of the first run to a file and compares later runs against the saved content to catch unintended regressions (in pytest, plugins such as syrupy provide this). It suits cases where the output is complex and writing the expected value by hand would be tedious, such as UI components or API responses.

# Conceptual implementation
import json
from pathlib import Path
from typing import Any
 
def assert_matches_snapshot(actual: Any, snapshot_path: Path) -> None:
    serialized = json.dumps(actual, indent=2, sort_keys=True, default=str)
 
    if snapshot_path.exists():
        stored = snapshot_path.read_text()
        if serialized != stored:
            raise AssertionError(
                f"Snapshot mismatch. Run with --update to update.\n"
                f"  Expected: {stored[:100]}...\n"
                f"  Got:      {serialized[:100]}..."
            )
    else:
        # First run: save the snapshot
        snapshot_path.parent.mkdir(parents=True, exist_ok=True)
        snapshot_path.write_text(serialized)

Code Coverage

Code coverage in Python is measured through trace hooks set with sys.settrace, or through sys.monitoring (Python 3.12+).

How coverage.py works:
  1. Sets a trace hook via sys.settrace()
  2. Python interpreter calls the hook on each line execution
  3. The hook records executed line numbers in a set
  4. After tests complete, compares executed lines against the executable statements to calculate coverage
 
Python 3.12+ sys.monitoring:
  1. sys.monitoring is a newer API with less overhead
  2. Monitors LINE events to record line execution
  3. coverage.py 7.x supports this new API

Watch Mode

Watch mode detects file changes and reruns tests automatically. pytest-watch implements this with filesystem monitoring via the watchdog library.

# Conceptual implementation
from watchdog.observers import Observer
from watchdog.events import FileSystemEventHandler
 
class TestRerunHandler(FileSystemEventHandler):
    def on_modified(self, event):
        if event.src_path.endswith(".py"):
            print(f"\nFile changed: {event.src_path}")
            run_tests()
 
def watch_mode(root_dir: str) -> None:
    print("Watching for changes...")
    observer = Observer()
    observer.schedule(TestRerunHandler(), root_dir, recursive=True)
    observer.start()
    observer.join()

Design Comparisons Across Frameworks

pytest vs unittest

pytest:
  - Discovery: filename + function name conventions (test_ prefix)
  - Assertions: built-in assert rewriting (AST transformation)
  - Fixtures: decorators + dependency injection via argument names
  - Plugins: pluggy-based plugin system
  - Parallelization: pytest-xdist (execnet)
 
unittest:
  - Discovery: test* methods on TestCase subclasses
  - Assertions: assertEqual, assertTrue method family
  - Fixtures: setUp / tearDown methods
  - Plugins: TestRunner customization
  - Parallelization: not supported out of the box
 
The key difference:
  pytest made "using plain assert statements" possible
  through AST rewriting. This eliminated the need for
  verbose method calls like unittest's
  self.assertEqual(a, b).

pytest Fixtures — Dependency Injection by Argument Name

pytest fixtures are functions that prepare the data or environment a test needs. pytest matches the argument names of a test function against fixture names and injects the matching fixtures automatically. This applies the idea behind dependency injection (DI) containers in web frameworks to tests. In a fixture that uses yield, the code before yield is setup and the code after it is teardown, so resource creation and cleanup are written in one place.

# pytest: fixtures are dependency injection
import pytest
 
@pytest.fixture
def database():
    """Create and destroy a test DB"""
    db = create_test_db()
    yield db        # ← inject db into the test
    db.cleanup()    # ← cleanup after the test
 
@pytest.fixture
def user(database):
    """A fixture that depends on the database fixture"""
    return database.create_user(name="Alice")
 
def test_insert(database):
    # Argument name "database" matches fixture name → auto-injected
    database.insert({"key": "value"})
    assert database.get("key") == "value"
 
def test_user_name(user):
    # user fixture auto-injected → database also auto-resolved internally
    assert user.name == "Alice"
 
# Parametric tests
@pytest.mark.parametrize("input_val,expected", [
    (1, 1),
    (2, 4),
    (3, 9),
])
def test_square(input_val, expected):
    assert input_val ** 2 == expected

pytest fixture resolution:
  1. Get test function argument names via inspect.signature
  2. Match argument names to fixture names
  3. Build a dependency graph (fixtures can depend on other fixtures)
  4. Determine execution order via topological sort
  5. Cache based on scope (function / class / module / session)

Go's testing Package

Go provides testing through the standard library's testing package and the go test command.

// Go: tests are identified by function signature conventions
func TestAdd(t *testing.T) {
    result := Add(1, 1)
    if result != 2 {
        t.Errorf("Add(1, 1) = %d, want 2", result)
    }
}
 
// Subtests — hierarchical structure similar to describe/it
func TestMath(t *testing.T) {
    t.Run("addition", func(t *testing.T) {
        if Add(1, 1) != 2 {
            t.Fatal("1+1 should be 2")
        }
    })
    t.Run("division by zero", func(t *testing.T) {
        defer func() {
            if r := recover(); r == nil {
                t.Fatal("expected panic")
            }
        }()
        Divide(1, 0)
    })
}

Go testing characteristics:
  - Discovery: `cmd/go` generates a test binary from `Test*` / `Benchmark*` / `Fuzz*` functions
  - Assertions: none in standard library (if + t.Error is the norm)
  - Parallelization: opt-in via t.Parallel()
  - Benchmarking: integrated via Benchmark* functions
  - Fuzzing: integrated via Fuzz* functions (Go 1.18+)
  - Testing concurrent code: virtual time via testing/synctest (Go 1.25+)

That last entry, testing/synctest, addresses a common problem with concurrency tests: waiting with time.Sleep makes them slow and flaky. synctest.Test runs the test function inside an isolated "bubble" where the time package uses a fake clock. When every goroutine in the bubble is blocked, the clock advances immediately, so a "one hour later" timeout can be tested deterministically in zero wall-clock time. It first appeared behind GOEXPERIMENT=synctest in Go 1.24 and graduated to general availability in Go 1.25.

Jest / Vitest

Jest and Vitest are two major test frameworks in the JavaScript / TypeScript ecosystem.

Jest:
  - Discovery: defaults equivalent to `**/__tests__/**/*.[jt]s?(x)` and `**/?(*.)+(spec|test).[tj]s?(x)` patterns
  - Collection: describe/it callback registration pattern
  - Assertions: expect().toBe() Fluent API
  - Mocking: jest.mock() automatic module mocking
  - Parallelization: child_process (fork) by default
 
Vitest:
  - Jest-compatible API but leverages the Vite ecosystem
  - Native ESM support
  - parallelization through worker threads / child processes depending on pool configuration

Summary

The mechanisms of each phase and their corresponding design patterns are as follows.

Phase	Mechanism	Design Pattern
Discovery	Glob patterns for file search	Convention over Configuration
Collection	Module import + inspect	Reflection
Lifecycle	setup/teardown hooks	Hook / Callback
Execution	try-except test isolation	Defensive Programming
Assertion	assert raises exceptions	AST Rewriting
Mocking	Function wrapping + call recording	Proxy Pattern
Reporting	Reporter abstract class	Strategy Pattern
Parallelization	execnet (process parallelism)	Message Passing

Knowing how these mechanisms work makes it easier to decide at what scope to use hooks and fixtures, and to tell at which stage a failure happened. Next time a test goes red, picture the assert statement running raise AssertionError and the runner's try-except catching it.