How we build, what we trust, what we won't do.

Each benchmark measures the three core operations of every algorithm — for KEMs that's key generation, encapsulation, and decapsulation; for signatures it's key generation, signing, and verification. Every operation runs for 1,000 timed iterations, preceded by 50 untimed warmup iterations.

Timing uses time.perf_counter_ns() with the garbage collector disabled and the process pinned to a single CPU core for the duration of measurement, so scheduler noise and GC pauses don't corrupt the measurements. For each operation we record mean, median, p95, p99, standard deviation, min, max, and derived throughput (operations per second), plus the relevant artifact sizes — public key, private key, ciphertext or signature length in bytes.

Every run also captures its own runtime conditions in a runtime_metrics block: wall-clock duration, CPU steal-time (delta of Linux kernel steal jiffies across the timed loop), and load averages at start and end. On burstable cloud instance classes this is essential — non-zero sustained steal-time signals throttling that would otherwise silently slow timed iterations. Healthy runs show near-zero steal. If a run was throttled, the audit trail shows it.

The current runner is an AWS EC2 t3.medium in us-east-1: a burstable instance class with CPU performance guaranteed at a baseline and the ability to burst above when CPU credits are available. Under sustained load with depleted credits, the instance throttles to baseline — which would silently slow timed iterations unless the audit trail catches it.

The earlier plan called for a fixed-performance c7i.large migration. We deferred that to Month 3 once the runtime_metrics block showed steal-time consistently near zero (around 0.24% on representative runs), meaning the burstable instance is running clean. Migration will happen when budget or load justifies it; when it does, this document will be updated, historical runs from the burstable period will remain available, and the hardware change will be explicitly dated. Results will not be silently migrated.

Access to the runner is through AWS SSM Session Manager (no port 22 open to the public internet). The runner itself is registered as a systemd-managed self-hosted GitHub Actions runner so it survives reboots and reconnects automatically.

The benchmark workflow at .github/workflows/benchmark.yml has two triggers only: scheduled (daily at 06:00 UTC) and manual workflow_dispatch. There is deliberately no pull_request trigger — running expensive measurement jobs on untrusted PR code is exactly the kind of supply-chain mistake we won't make. A concurrency group on the workflow serializes runs so a second trigger queues behind the first rather than racing on result-file writes.

permissions: is scoped to contents: write at the workflow level (least privilege; we only need to commit the result file back). The bot's commit message includes [skip ci] to prevent the workflow from re-triggering on its own commits.

Each run writes benchmark/results/results-YYYY-MM-DD-{short_sha}.json — every run is its own file, named by date and the git commit SHA the benchmark was running. The dashboard reads the full series so history is preserved on disk and every chart point has a stable identity. The loader handles both this filename pattern and the older results-YYYY-MM-DD.json pattern from early Week 2 for backward compatibility.

liboqs 0.15.0, built from source with -DOQS_DIST_BUILD=ON for runtime CPU dispatch. liboqs-python is pinned to the matching 0.15.0 in a Python 3.12 virtual environment to avoid the version-skew warnings that older mismatched pairs produce. The exact versions for the latest run appear in the “Environment” table above and in every result file's environment block.

A naming subtlety worth knowing: SLH-DSA variants in liboqs 0.15+ are exposed as SLH_DSA_PURE_SHAKE_128S and SLH_DSA_PURE_SHAKE_128F — the PURE prefix and underscored form differ from earlier liboqs releases and from some documentation. The benchmark script uses these exact identifiers.

Three patterns emerge when Q-Shield results are compared against eBACS/SUPERCOP reference cycle counts for the same liboqs 0.15.0 primitives. Understanding them is necessary to read the numbers correctly.

ML-KEM: measured faster than eBACS reference (+24–27%). ML-KEM-768 decapsulation measures ~48 μs on the production box, faster than X25519 (~161 μs). This is real: AVX2-dispatched lattice arithmetic genuinely outperforms elliptic-curve scalar multiplication for this operation, confirmed by liboqs reference data. The +24–27% delta versus eBACS is a methodology difference: our harness reuses pre-allocated key material across iterations, whereas the liboqs speed_kem tool includes fresh keygen in its timing loop. We measure the encapsulation and decapsulation operations in isolation; eBACS includes keygen overhead. Net result: our Python harness, despite its binding overhead, reports lower latency for encaps/decaps than the eBACS figure.

ML-DSA and Falcon: measured slower than eBACS reference (−28 to −36%). ML-DSA-44 sign median is 105 μs; Falcon-512 sign median is 343 μs. Two causes compound. First, our harness reuses a fixed message across iterations; the liboqs speed_sig tool calls OQS_randombytes() per iteration, inflating its reported sign time with randomness generation cost. Our times should therefore be lower than eBACS — the fact that they are higher reveals the second cause: t3.medium burstable CPU throttling. Under the sustained sequential load of 1,000-iteration loops, CPU steal time degrades throughput and the 2.50 GHz nominal clock is not reliably available. This manifests as elevated sign times and high p99 variance (ML-DSA-44 sign p99: 416 μs vs. median 105 μs — a 4× spread). Sign operations are more throttle-sensitive than verify because they are compute-heavier. Verify times and keygen times are reliable on the production box; sign times for lattice schemes should be read as upper bounds.

SLH-DSA: tight cross-validation (±1%). SLH-DSA-SHAKE-128S sign median is 1,289 ms on the production box — slow by design for the small-signature parameter set. The tight agreement with eBACS is expected: SLH-DSA is dominated by Keccak/SHA3 hashing, and the SHA3 path on this host dispatches to AVX512VL (OQS_USE_SHA3_AVX512VL=1, confirmed in oqsconfig.h) — the same well-optimised route eBACS exercises. Hashing is also less sensitive to burstable throttling than lattice operations. The 128F (fast) variant signs in ~61 ms at the cost of a larger signature (17 KB vs. 7.9 KB for 128S).

The planned migration to a c7i.large dedicated-core instance will eliminate the burstable-throttle caveat on lattice sign times. Results before and after migration will be annotated as a measurement-environment change, not treated as a continuous series.

The composed-protocol tracks (tls_composed.py, ssh_composed.py) measure real hybrid post-quantum suites (X25519MLKEM768, SecP256r1MLKEM768, SSH's mlkem768x25519-sha256) alongside their classical baselines (X25519, curve25519-sha256) — this data has been committed daily since June 2026; the gap was that it never surfaced on /q-shield/compare, only on /q-shield/protocols. Both pages now expose it.

Amplification factor — bytes the server returns per byte the client sends, per suite — is computed purely from the size.bytes_client_to_server and size.bytes_server_to_client fields already in every composed-suite record (see web/lib/protocols/derive.ts). No new benchmarking; it is the number a DoS-conscious architect wants and, as far as this project has found, is not published elsewhere for these suites.

What “bytes on the wire” means here, precisely. The composed-suite bytes_total figure is the cryptographic key-exchange payload — public key, ciphertext, classical key share — not a full captured TLS record (no ClientHello extensions, no certificate chain, no record-layer framing). Every UI surface labels this “key-exchange payload bytes,” not “handshake bytes on the wire” unqualified, so the number isn't read as more complete than it is.

AES-256-GCM baseline and LMS/XMSS stateful signatures (benchmark/protocols/aes_baseline.py, benchmark/protocols/lms_xmss.py) exist as harness code but are not yet wired into the daily workflow — see each script's own docstring for why, and for LMS/XMSS specifically, for the ephemeral-keypair design that keeps a stateful signature scheme safe to benchmark at all. Any table or chart showing LMS/XMSS carries a persistent “stateful — firmware/code-signing use case, not general TLS” label, not a hover tooltip, and shows “queued” rather than a fabricated number until a real run exists.

Handshake packet count (and whether it exceeds initcwnd), connections per core, and bytes of state per half-open connection are not measured by anything this repo currently runs. The composed-protocol tracks above time cryptographic operations and accounts key/ciphertext sizes in-process — it does not open a socket or capture packets. Producing these numbers for real would require Layer B (live handshakes, currently planned via Docker), which does not exist yet. This is stated here explicitly rather than left as a silent gap, per the same sourcing discipline that governs every other claim on this page: an absence is a fact worth stating, not a placeholder worth guessing at.

The target the score is computed against is the Gidney 2025 estimate: “How to factor 2048-bit RSA integers with less than a million noisy qubits” — roughly 1,000,000 physical qubits, under a week of runtime, to factor RSA-2048 with Shor's algorithm. This is a peer-reviewed resource estimate, not a law of physics, and it is the most important caveat on the entire dashboard.

The target fell roughly 20× in six years on algorithmic and error-correction progress alone. The 2019 Gidney & Ekera paper estimated 20 million qubits over 8 hours; the 2025 paper does it in under a million, under a week, with hardware assumptions held identical. The scores on the dashboard are distance to that 2025 figure today. If the target moves again — and it will — every score moves with it, and the new anchor is cited.

The scoring formula is built so the anchor can be swapped (for Bitcoin/ECDSA-256, RSA-3072, TLS, etc.) by changing one numeric constant. The launch target is RSA-2048 only.

The score is computed as a multiplicative gate:

LogicalCapacity is how many standing, error-corrected logical qubits the system has divided by what the anchor demands. Today this is zero for every scored system — there are no standing, always-on logical qubits anywhere — so the entire product zeroes out. That is the honest answer for nearly all hardware.

FidelityGate is a 0-to-1 multiplier on whether the system's two-qubit gate fidelity clears the fault-tolerance threshold required for that system's error-correction code to actually suppress errors. Below threshold, more qubits don't help — they add more errors than they correct.

ECSignal rewards systems that have publicly demonstrated a below-threshold error-correction result (the next section). Until a system has shown error suppression scales with code distance, it has no demonstrated path to standing logical qubits, and the multiplier reflects that.

The multiplicative structure is the point. Any of the three components going to zero takes the score to zero. There is no “mostly there” halfway state where a machine with great qubits but no error correction earns partial credit. Breaking RSA requires all of these; the score reflects all of them.

A system earns the ECSignal multiplier only with a publicly reported demonstration that adding error-correction code distance reduces logical error rate — the fundamental below-threshold result. This is a rule, not editorial: the score reflects what has been demonstrated, not what is plausible or claimed.

At the time of writing, one system in the dataset clears this bar. That system is not named in the hero of the dashboard — the table names it. The frontier is non-zero today only because of this rule; as other systems clear the same bar, the ranking will shift accordingly.

The hero number on the dashboard is the field frontier — the highest threat score currently held by any scored machine — not a named winner. The reason is editorial hygiene: leading with “Vendor X is closest to breaking RSA” would be equivalent to crowning a winner, which is incompatible with neutrality. The per-system table beneath the hero names every machine and its score. The reader concludes; we don't.

The threat score is uncompromising: until a system has standing error-corrected logical qubits, the score is zero. But that creates a problem — the dashboard would show only one non-zero score and look static. So the dashboard also surfaces a separate readiness axis, with its own visual treatment (stacked bars, never a gauge), showing how far each system has progressed through the preconditions: fidelity progress, error-correction progress, scale progress.

Readiness is not the threat score. It does not measure distance to breaking RSA. It measures preconditions assembled. A system can have 80% readiness and zero threat at the same time, and that is exactly the situation for most of the field today. The two are deliberately rendered as structurally different visuals (gauge vs stacked bars) so they can't be confused.

Every numeric input to the score must come from one of:

• A peer-reviewed publication (arXiv preprints accepted to a venue, journal of record)
• An official vendor technical document or specification page

Press releases, blog posts, secondary aggregators, and content-mill restatements do not clear the bar. Where a published value differs from a press figure, the published value wins and the gap is noted on the row — for example, Google's blog says coherence is “approaching 100 µs,” the Nature paper says 68 µs; the dataset cites the paper. When the bar fails for a given system (Atom Computing's next-generation array hasn't shipped per-system T1/T2, for example), the value is null and a note explains the omission. Null is an honest value.

Every numeric field also carries: its source URL, its confidence level (peer-reviewed vs vendor-published), its measurement method (XEB vs median ECR error vs randomized benchmarking vs direct RB, etc.; see glossary below), and the date the figure was true.

The raw two-qubit gate fidelity column on the dashboard is deliberately not displayed as a ranking. The reason: a 99.4% from Google measured via cross-entropy benchmarking (XEB) is not the same physical quantity as a 99.2% from IBM measured via median ECR error, which is not the same as a 99.8% from Quantinuum measured via randomized benchmarking on a trapped-ion gate. They are different measurement protocols on different physical hardware and they are not comparable to the third decimal.

Each fidelity value on the dashboard carries its measurement method as a tag (hover the tag in any expanded row to see the definition; see also the glossary below). The scoring formula treats fidelity through the FidelityGate term against the fault-tolerance threshold, not as a ranked comparison across vendors — which is the only way to use these numbers honestly.

Analog Hamiltonian simulators — QuEra Aquila, Pasqal Orion Alpha — have no gate-model two-qubit fidelity in the discrete-gate sense, and no gate-model pathway to factoring RSA in the launch scoring sense. They are not bad gate-model machines; they are a different category of machine, useful for different physics problems.

The score for these systems is shown as explicit N/A, never as a low numeric score. Conflating “category mismatch” with “low performance” would be a methodological error.

Each scored system carries an aggregate input-confidence rating shown on the row:

• High: all required fields sourced to peer-reviewed publications.
• Medium: a mix of peer-reviewed and vendor-published sources.
• Low: primarily vendor-published, with one or more fields lacking a peer-reviewed citation.

The rating is mechanical — it's computed from the per-field confidence tags, not hand-assigned. A high score on a low-confidence row should be read with appropriate skepticism.

No quantum hardware vendor pays for placement, ranking, ordering, or early access on the Q-Day Index. The dataset is independent. The hero is the field frontier, not a named machine. The per-system table presents every system at the same depth, with every input sourced. If we ever take vendor sponsorship of any kind, it will be announced publicly, with the terms in writing, and the affected rows will be flagged.

We do not publish a projected Q-Day year. A projection is only as good as its underlying trajectory model, and we have not yet built one that survives hostile inspection. The historical entries in the dataset (Sycamore 2019, Eagle 2021, Condor 2023) exist so that a future trajectory model has data to fit, but the model itself is held internal until it is defensible. When we publish a projected year, the model, the assumptions, and the inputs will all be open at the same time. Anything else would be a guess dressed up as analysis.

Every measurement is an ordinary TLS handshake with a publicly-advertised hostname on a standard port — the same handshake any browser completes visiting the page. We record only the negotiated protocol parameters the server voluntarily presents: highest negotiated TLS version, whether TLS 1.2 is still accepted, which key-exchange groups the server accepts when offered, and the certificate chain's metadata (signature algorithm, public key algorithm, key size, issuer, validity dates).

We never authenticate, never scan for vulnerabilities, never touch a non-standard port, and never parse or store page content. If an endpoint blocks, rate-limits, or refuses a connection, that is the answer — it is recorded as not measurable and we move on. We do not work around it.

Key-exchange group support is measured by connecting once per candidate group and observing whether the server accepts it — not by reading which group the server happens to negotiate by default when several are offered. “Accepts X25519MLKEM768 when it's the only group offered” and “prefers it in a normal handshake” are different claims. Conflating them is the first correction request we would receive, and would deserve.

KEX is the share of an institution's measurable endpoints that accept at least one hybrid post-quantum key-exchange group (X25519MLKEM768, SecP256r1MLKEM768). AUTH is the share presenting a post-quantum or hybrid certificate signature. They are reported side by side, always, never combined into one number.

The reason is the finding the whole index exists to surface: as of this methodology version, ML-DSA is not permitted in publicly-trusted TLS certificates at all — the CA/Browser Forum's Baseline Requirements (v2.2.9 §6.1.5) permit only RSA and ECDSA key pairs, full stop. AUTH is at or near zero across the entire index today, by construction, while KEX moves. A single composite score would average those two facts into a misleading middle, exactly hiding the finding.

An institution with zero measurable endpoints (every endpoint blocked, unreachable, or excluded) shows “—” in both tracks, never 0%. Those are different claims — one is “we don't know,” the other is a specific, false “we know, and the answer is none.”

Four flags are recorded and displayed on every row, independent of the two-track score: whether TLS 1.2 is still accepted, a weak or deprecated certificate signature algorithm, a certificate key below the recommended minimum size, and a certificate that is expired or within 90 days of expiry. These are not scored — they are shown because they are independently worth knowing, and folding them into a score would obscure which specific thing is wrong.

Negotiated parameters depend on the client as well as the server — the measurement paper this index builds on notes this explicitly. Every scan run publishes its exact client configuration (OpenSSL version, offered group list, candidate groups probed) as a hash stamped on every record. A configuration change is a methodology-version change, dated and disclosed, never a silent shift in what a score means.

Institutions are drawn from public regulator and association listings — the Financial Stability Board's G-SIB list, national bank-holding-company and systemic-institution lists (US Federal Reserve, EU EBA O-SII list, UK ring-fencing regime), the CA/Browser ecosystem's own trusted-root reports, and well-known card networks — never scraped or crawled. Endpoints (the specific hostnames measured per institution) are discovered only from what the institution has itself published: linked login pages, announced developer portals, documented mobile endpoints. No endpoint is guessed from a naming convention and left unverified.

At most one connection per second to any single hostname; at most one concurrent connection to any one institution; a full sweep once per week. Any connection refusal or reset ends measurement of that host for the entire sweep immediately — not just the probe that hit it. The scanning source is a single, stable, publicly-disclosed address with correct reverse DNS and a monitored abuse contact, so any institution that notices the traffic can identify it and request exclusion.

Any institution may request exclusion from measurement, unconditionally, without justification. An excluded institution is not silently removed from the index and is not scored as zero — its row states plainly “excluded at the institution's request,” dated, with no score in either track. Silent removal would let the index be edited by whoever complains; scoring an exclusion as zero would make opting out punitive. Naming the exclusion is the only version that is neither.

Every institution sees its own complete data — every endpoint, every negotiated parameter, both track scores — before its row is ever published for the first time, with 14 calendar days to respond. Material changes (a track score crossing a published threshold, a new hygiene flag) trigger a shorter notice window before the change goes live. Silence is not consent and is not a veto — after the window elapses without response, publication proceeds with a note that reply was invited.

A permanent, published address plus a correction link on every row accepts reports from anyone, not only the rated institution. Every report is acknowledged within 5 business days; a disputed row is marked disputed publicly while under review. A confirmed correction updates the row, carries a visible correction notice for one full refresh cycle, and gets a dated entry in a permanent public corrections log — a wrong number is never silently fixed.

Some measured posture reflects a CDN or edge provider in front of an institution rather than the institution's own infrastructure — “your PQ readiness is rented” is itself part of the finding, per the spec this index is built from. Automatic CDN detection and per-row attribution is specified but not yet built; until it is, a strong key-exchange score should be read with that caveat in mind, not as a claim about the institution's own stack.

Rated institutions may buy data, reports, and analyst time. They receive no influence over methodology, scoring, their own position, or publication timing. No pre-publication access exists beyond right of reply, identical for customers and non-customers. We do not sell migration tools or remediation services to a rated institution — selling the fix for the problem we score would end the only thing that makes the score worth reading.