How we build, what we trust, what we won't do.

Each benchmark measures the three core operations of every algorithm — for KEMs that's key generation, encapsulation, and decapsulation; for signatures it's key generation, signing, and verification. Every operation runs for 1,000 timed iterations, preceded by 50 untimed warmup iterations.

Timing uses time.perf_counter_ns() with the garbage collector disabled and the process pinned to a single CPU core for the duration of measurement, so scheduler noise and GC pauses don't corrupt the measurements. For each operation we record mean, median, p95, p99, standard deviation, min, max, and derived throughput (operations per second), plus the relevant artifact sizes — public key, private key, ciphertext or signature length in bytes.

Every run also captures its own runtime conditions in a runtime_metrics block: wall-clock duration, CPU steal-time (delta of Linux kernel steal jiffies across the timed loop), and load averages at start and end. On burstable cloud instance classes this is essential — non-zero sustained steal-time signals throttling that would otherwise silently slow timed iterations. Healthy runs show near-zero steal. If a run was throttled, the audit trail shows it.

The current runner is an AWS EC2 t3.medium in us-east-1: a burstable instance class with CPU performance guaranteed at a baseline and the ability to burst above when CPU credits are available. Under sustained load with depleted credits, the instance throttles to baseline — which would silently slow timed iterations unless the audit trail catches it.

The earlier plan called for a fixed-performance c7i.large migration. We deferred that to Month 3 once the runtime_metrics block showed steal-time consistently near zero (around 0.24% on representative runs), meaning the burstable instance is running clean. Migration will happen when budget or load justifies it; when it does, this document will be updated, historical runs from the burstable period will remain available, and the hardware change will be explicitly dated. Results will not be silently migrated.

Access to the runner is through AWS SSM Session Manager (no port 22 open to the public internet). The runner itself is registered as a systemd-managed self-hosted GitHub Actions runner so it survives reboots and reconnects automatically.

The benchmark workflow at .github/workflows/benchmark.yml has two triggers only: scheduled (daily at 06:00 UTC) and manual workflow_dispatch. There is deliberately no pull_request trigger — running expensive measurement jobs on untrusted PR code is exactly the kind of supply-chain mistake we won't make. A concurrency group on the workflow serializes runs so a second trigger queues behind the first rather than racing on result-file writes.

permissions: is scoped to contents: write at the workflow level (least privilege; we only need to commit the result file back). The bot's commit message includes [skip ci] to prevent the workflow from re-triggering on its own commits.

Each run writes benchmark/results/results-YYYY-MM-DD-{short_sha}.json — every run is its own file, named by date and the git commit SHA the benchmark was running. The dashboard reads the full series so history is preserved on disk and every chart point has a stable identity. The loader handles both this filename pattern and the older results-YYYY-MM-DD.json pattern from early Week 2 for backward compatibility.

liboqs 0.15.0, built from source with -DOQS_DIST_BUILD=ON for runtime AVX2 detection. liboqs-python is pinned to the matching 0.15.0 in a Python 3.12 virtual environment to avoid the version-skew warnings that older mismatched pairs produce. The exact versions for the latest run appear in the “Environment” table above and in every result file's environment block.

A naming subtlety worth knowing: SLH-DSA variants in liboqs 0.15+ are exposed as SLH_DSA_PURE_SHAKE_128S and SLH_DSA_PURE_SHAKE_128F — the PURE prefix and underscored form differ from earlier liboqs releases and from some documentation. The benchmark script uses these exact identifiers.

The target the score is computed against is the Gidney 2025 estimate: “How to factor 2048-bit RSA integers with less than a million noisy qubits” — roughly 1,000,000 physical qubits, under a week of runtime, to factor RSA-2048 with Shor's algorithm. This is a peer-reviewed resource estimate, not a law of physics, and it is the most important caveat on the entire dashboard.

The target fell roughly 20× in six years on algorithmic and error-correction progress alone. The 2019 Gidney & Ekera paper estimated 20 million qubits over 8 hours; the 2025 paper does it in under a million, under a week, with hardware assumptions held identical. The scores on the dashboard are distance to that 2025 figure today. If the target moves again — and it will — every score moves with it, and the new anchor is cited.

The scoring formula is built so the anchor can be swapped (for Bitcoin/ECDSA-256, RSA-3072, TLS, etc.) by changing one numeric constant. The launch target is RSA-2048 only.

The score is computed as a multiplicative gate:

LogicalCapacity is how many standing, error-corrected logical qubits the system has divided by what the anchor demands. Today this is zero for every scored system — there are no standing, always-on logical qubits anywhere — so the entire product zeroes out. That is the honest answer for nearly all hardware.

FidelityGate is a 0-to-1 multiplier on whether the system's two-qubit gate fidelity clears the fault-tolerance threshold required for that system's error-correction code to actually suppress errors. Below threshold, more qubits don't help — they add more errors than they correct.

ECSignal rewards systems that have publicly demonstrated a below-threshold error-correction result (the next section). Until a system has shown error suppression scales with code distance, it has no demonstrated path to standing logical qubits, and the multiplier reflects that.

The multiplicative structure is the point. Any of the three components going to zero takes the score to zero. There is no “mostly there” halfway state where a machine with great qubits but no error correction earns partial credit. Breaking RSA requires all of these; the score reflects all of them.

A system earns the ECSignal multiplier only with a publicly reported demonstration that adding error-correction code distance reduces logical error rate — the fundamental below-threshold result. This is a rule, not editorial: the score reflects what has been demonstrated, not what is plausible or claimed.

At the time of writing, one system in the dataset clears this bar. That system is not named in the hero of the dashboard — the table names it. The frontier is non-zero today only because of this rule; as other systems clear the same bar, the ranking will shift accordingly.

The hero number on the dashboard is the field frontier — the highest threat score currently held by any scored machine — not a named winner. The reason is editorial hygiene: leading with “Vendor X is closest to breaking RSA” would be equivalent to crowning a winner, which is incompatible with neutrality. The per-system table beneath the hero names every machine and its score. The reader concludes; we don't.

The threat score is uncompromising: until a system has standing error-corrected logical qubits, the score is zero. But that creates a problem — the dashboard would show only one non-zero score and look static. So the dashboard also surfaces a separate readiness axis, with its own visual treatment (stacked bars, never a gauge), showing how far each system has progressed through the preconditions: fidelity progress, error-correction progress, scale progress.

Readiness is not the threat score. It does not measure distance to breaking RSA. It measures preconditions assembled. A system can have 80% readiness and zero threat at the same time, and that is exactly the situation for most of the field today. The two are deliberately rendered as structurally different visuals (gauge vs stacked bars) so they can't be confused.

Every numeric input to the score must come from one of:

• A peer-reviewed publication (arXiv preprints accepted to a venue, journal of record)
• An official vendor technical document or specification page

Press releases, blog posts, secondary aggregators, and content-mill restatements do not clear the bar. Where a published value differs from a press figure, the published value wins and the gap is noted on the row — for example, Google's blog says coherence is “approaching 100 µs,” the Nature paper says 68 µs; the dataset cites the paper. When the bar fails for a given system (Atom Computing's next-generation array hasn't shipped per-system T1/T2, for example), the value is null and a note explains the omission. Null is an honest value.

Every numeric field also carries: its source URL, its confidence level (peer-reviewed vs vendor-published), its measurement method (XEB vs median ECR error vs randomized benchmarking vs direct RB, etc.; see glossary below), and the date the figure was true.

The raw two-qubit gate fidelity column on the dashboard is deliberately not displayed as a ranking. The reason: a 99.4% from Google measured via cross-entropy benchmarking (XEB) is not the same physical quantity as a 99.2% from IBM measured via median ECR error, which is not the same as a 99.8% from Quantinuum measured via randomized benchmarking on a trapped-ion gate. They are different measurement protocols on different physical hardware and they are not comparable to the third decimal.

Each fidelity value on the dashboard carries its measurement method as a tag (hover the tag in any expanded row to see the definition; see also the glossary below). The scoring formula treats fidelity through the FidelityGate term against the fault-tolerance threshold, not as a ranked comparison across vendors — which is the only way to use these numbers honestly.

Analog Hamiltonian simulators — QuEra Aquila, Pasqal Orion Alpha — have no gate-model two-qubit fidelity in the discrete-gate sense, and no gate-model pathway to factoring RSA in the launch scoring sense. They are not bad gate-model machines; they are a different category of machine, useful for different physics problems.

The score for these systems is shown as explicit N/A, never as a low numeric score. Conflating “category mismatch” with “low performance” would be a methodological error.

Each scored system carries an aggregate input-confidence rating shown on the row:

• High: all required fields sourced to peer-reviewed publications.
• Medium: a mix of peer-reviewed and vendor-published sources.
• Low: primarily vendor-published, with one or more fields lacking a peer-reviewed citation.

The rating is mechanical — it's computed from the per-field confidence tags, not hand-assigned. A high score on a low-confidence row should be read with appropriate skepticism.

No quantum hardware vendor pays for placement, ranking, ordering, or early access on the Q-Day Index. The dataset is independent. The hero is the field frontier, not a named machine. The per-system table presents every system at the same depth, with every input sourced. If we ever take vendor sponsorship of any kind, it will be announced publicly, with the terms in writing, and the affected rows will be flagged.

We do not publish a projected Q-Day year. A projection is only as good as its underlying trajectory model, and we have not yet built one that survives hostile inspection. The historical entries in the dataset (Sycamore 2019, Eagle 2021, Condor 2023) exist so that a future trajectory model has data to fit, but the model itself is held internal until it is defensible. When we publish a projected year, the model, the assumptions, and the inputs will all be open at the same time. Anything else would be a guess dressed up as analysis.

Q-Arena will run standardised quantum circuits — Bell-state preparation and small entanglement-witness tests initially — across multiple machines, so machines can be compared on what they actually do rather than what their data sheets claim. Targets are AWS Braket simulators and the IBM Quantum free tier for real hardware. For each (circuit, machine) pair, we will record success rate, gate count, queue time, and circuit depth, and compare real-hardware results against simulator-ideal so noise can be isolated from algorithmic structure.

Q-Arena is not yet running. This section is a stub describing intent. Exact circuit list, machine roster, and run cadence will be finalised as we begin running them. Anything stated here in advance is a plan, not a promise; this section will be updated when the first runs ship.