Benchmarking guide¶

Calibration¶

BenchCore starts with one iteration and grows the count until the measured round reaches target_round_time_ns. Growth is bounded by max_iterations and the cooperative max_calibration_time_ns budget.

Calibration samples are discarded. They do not appear in warmup, measured rounds, statistics, or reports.

Use fixed iterations when the callable mutates inputs, consumes a finite iterator, changes cost between calls, or must reproduce an earlier run exactly.

Lifecycle hooks¶

Setup and teardown execute around every calibration, warmup, and measured round:

from benchcore.core.models import BenchmarkConfig
from benchcore import Bench

values = [3, 2, 1]


def reset() -> None:
    values[:] = [3, 2, 1]


bench = Bench(
    BenchmarkConfig(iterations=1),
    round_setup=reset,
    round_teardown=values.clear,
)
result = bench.run(values.sort)

Hooks are outside the timed interval. Teardown runs once after an active round, including when the callable or timer fails.

Statistics and units¶

Stored whole-region durations use integer nanoseconds. Per-iteration durations and statistics use floats because normalization can produce fractions of a timer tick.

standard_deviation_ns is the sample standard deviation across measured rounds and is 0.0 for one round. These are descriptive statistics, not evidence of statistical significance.

Responsible comparison¶

Control CPU load, power management, inputs, interpreter build, and environment. BenchCore rejects built-in comparisons when report names or environment identity differ, but environment equality alone does not make a benchmark scientifically controlled.