How we measure — and where our own numbers are soft.

Everything we publish is meant to be checkable. That’s easy to say and slightly awkward to live with, because it means publishing the conditions a number was produced under even when those conditions are unflattering. This week they were. That part is at the bottom, and it’s the most useful section here.

Three pillars

Every benchmark, public. Source code, test parameters, scoring engine, dataset and full result sets live in a public repository. No paywall, no NDA, no proprietary harness. If the harness is secret, the number is an assertion.

Every run, auditable. Benchmarks execute on a schedule, daily, on dedicated hardware. Every workflow log is public, every result commit is timestamped, and every data point on the dashboard links back to the run that produced it.

Every score, reproducible. The Q-Day Index scoring engine is deterministic against the committed dataset — the same inputs produce identical scores to the third decimal. Q-Shield numbers reproduce within run-to-run variance on equivalent hardware. If you can’t reproduce one, that’s a bug worth filing, and we’d rather have the issue than the benefit of the doubt.

The run protocol

Q-Shield measures wall-clock performance of the NIST-standardized algorithms: key generation, encapsulation and decapsulation for KEMs; key generation, signing and verification for signatures. Coverage is FIPS 203 (ML-KEM), FIPS 204 (ML-DSA) and FIPS 205 (SLH-DSA), measured through liboqs 0.15.0 with liboqs-python pinned to the matching version.

Each operation runs 1,000 timed iterations after 50 untimed warmups, with:

• timing via time.perf_counter_ns(),
• the garbage collector disabled,
• the process pinned to a single CPU core.

We record mean, median, p95, p99, standard deviation, min, max and operations per second — not just an average, because the shape of the distribution is frequently the whole story. ML-DSA’s signing loop uses rejection sampling, so its worst case has no realistic upper bound; a mean alone would hide that entirely.

Each run writes its own file, named by date and commit SHA. Runs are never overwritten. The dashboard reads the full series, which is what makes drift visible instead of invisible.

The sourcing bar

For the Q-Day Index, every numeric input comes from a peer-reviewed publication or an official vendor technical document. Press releases and secondary aggregators don’t clear the bar, and where a press release conflicts with a published figure, the published figure wins. A concrete example: one vendor’s blog describes coherence as approaching 100µs while the corresponding Nature paper says 68µs. The dataset cites the paper.

Two related refusals, both deliberate. We don’t rank raw two-qubit gate fidelity, because a figure measured by cross-entropy benchmarking is not the same physical quantity as one measured by median ECR error, and putting them in a sorted column would imply otherwise. And we don’t publish a projected Q-Day year — a projection is only as good as its trajectory model, and we haven’t built one that survives hostile inspection. The historical entries exist so that a future model has something to fit; the model itself stays internal until it’s defensible.

Where our own numbers are soft

The benchmark box is an AWS t3.medium — a burstable instance. Burstable means the host can take CPU away from you under contention, which shows up as steal time. Every result file carries a runtime-metrics block recording steal time across the timed loop, precisely so throttling lands in the audit trail instead of silently corrupting a number.

Until recently that block read as reassurance: representative runs sat around 0.24% steal, which is clean, and the planned migration to a dedicated-core instance was deferred on that basis. Then it stopped being reassurance and started doing its actual job.

The honest reading is that timing deltas from this host are a distribution, not a verdict, and that the deferral of the instance migration was the wrong call — made on a steal-time figure that was true when it was measured and stopped being true later, which is the exact failure mode this whole publication exists to complain about. It is now scheduled rather than deferred.

What does not move: the byte counts. A hybrid X25519MLKEM768 handshake carries 2,336 bytes against classical X25519’s 64 regardless of how busy the host is, because that figure is determined by the protocol, not the processor. On the homepage and in the comparison views, the wire column is the durable number and the timing column is the noisy one. We’d rather tell you which is which than present them as equally solid.

The other limitations, stated up front

• liboqs is a prototyping library, not a production cryptographic implementation. Our numbers are representative of the algorithms; they should not be cited as authoritative for a specific production deployment.
• Hardware coverage is narrow. x86_64 and Graviton3 today. Broader silicon is a deliverable, not a claim.
• Q-Day Index depth is limited by how many vendors publish qualifying specs. Coverage grows as they do.

How to challenge a result

Open an issue or a pull request. Every Q-Shield run logs its commit, CPU model and full environment; every Q-Day Index figure carries its source URL, measurement method and the date it was true. If you can reproduce a Q-Shield discrepancy of more than two standard deviations on equivalent hardware, or you have a sourced correction to an index value, we want it — a corrections policy that never gets used usually means nobody is checking.

The canonical, always-current version of all of this lives at the methodology page. If this article and that page ever disagree, the page wins.