Software 4 Senses

← Learn

Measuring WebGPU Compute Honestly: an A/B Discipline

Software 4 Senses LLC · Last updated

GPU timing in a browser fails quietly: the default WebGPU clock is quantized, position within a submit biases readings by about 2% with an unstable sign, and a broken harness produces a number that looks exactly like a result. The remedy is a fixed protocol — characterize the instrument before trusting it, gate every comparison on byte-identical images, and keep negative results on permanent record. This page documents the standing discipline of the Software 4 Senses 3D Spatial Web Studio's rendering engine; it ports to any WebGPU compute workload.

Your instrument lies first

Chromium deliberately degrades the WebGPU timestamp clock, so the instrument itself has to be characterized before any measurement built on it can be believed. Chromium forces Dawn's timestamp_quantization device toggle on by default as a privacy mitigation, truncating every timestamp-query reading — raw timestamps and deltas alike — onto a 65,536 ns lattice. Launching the browser with --disable-dawn-features=timestamp_quantization restores raw 1 ns timestamps: with the flag, 24 of 24 repeated readings of the same workload were distinct; quantized, only 7 were. One machine, Apple silicon (Metal 3), Chromium-based test browser, ; single-machine result.

Timestamp quantization — a default-on privacy mitigation in Chromium's WebGPU implementation (Dawn) that truncates every GPU timestamp to a 65,536 ns lattice (mask 0xFFFF0000). The launch flag --disable-dawn-features=timestamp_quantization restores 1 ns resolution. Removing quantization does not remove within-submit position bias.

With the lattice removed, the clock characterizes well. On a synthetic ALU kernel, measured time scaled exactly 4.0× per 4× work across the first three steps (64/256/1024/4096 loop iterations → 0.197 / 0.786 / 3.146 / 10.289 ms); four passes encoded into one submit decomposed cleanly, the two identical heavy passes reading byte-identical; and GPU-timeline deltas summed to roughly 96% of wall-clock for the same submit. Same rig and date as above.

Two behaviors survive the flag and shape the whole protocol. First, within-submit position bias: the same workload in different timing slots of one submit read 0.0–1.9% apart, and the sign of the difference is not stable — it flipped between two runs an hour apart. So treat ~2% as the standing bias, and use the instability as the cheapest strong validity check available: a real effect must survive the bias reversing sign between independent runs. (An in-study claim that the quantization flag collapses this bias to 0.1% was refuted by the study's own data.) Second, warm-up: the first ~5 readings ran 12–41% high before a plateau with under 0.1% spread. Discard them.

Timestamps time passes, never subsystems inside one dispatch. That limitation becomes an asset when the kernel under test is a pure function from bound inputs to output pixels — no storage writes, no atomics. N idempotent dispatches of N shader configurations can then be encoded into a single submit, the configuration you intend to keep placed last. Every arm shares one queue, one thermal state, one page load — making cross-launch drift (previously observed at ±4–6 ms between page launches on this engine) structurally absent rather than statistically controlled for.

The eight-clause protocol

The standing A/B protocol has eight clauses, and a violation of any single one is grounds to void the run.

  1. Pin one engine, one scene, one pose — before settling. Pause the render loop first, then let the scene settle. An unpinned orbit pose was measured to swing dark-pixel counts about 20% run to run on the same scene. Pinned-rig traps recorded during the engine's rendering-artifact hunt.
  2. Byte-for-byte pixel identity before any timing. A faster arm that renders differently is an arm rendering less; compare its output bytes to the baseline first, and only time arms that match exactly.
  3. Mirrored A-B-B-A slot ordering, per-position medians. The mirrored order cancels within-submit position bias instead of modeling it; per-position medians keep any residual bias visible.
  4. Discard warm-up rounds; medians, never means. The characterized instrument runs its first ~5 readings 12–41% high; a mean lets those outliers vote.
  5. Publish a noise floor with every result — same configuration in every timing slot, same machine, same day. A result quoted without its floor is a claim without a resolution limit.
  6. Interleave floor ladders; never subtract a one-time bias correction. Run a floor before and after each arm and let their spread define the run's resolution band. A bias estimated once, under the very noise it is meant to remove, is not a correction (see the next section for what happens when this clause is skipped).
  7. Effects inside the floor band are nulls. Empirically the band ran 1.0–4.5% depending on how much saturated GPU work the session had already done — prefer fewer arms per session; anything inside the band is a null, not a small win.
  8. Cold-compile numbers use cache-busted pipelines. Compiled shaders are cached below the page, keyed on the WGSL text, and on the test machine the cache outlived even a fresh browser profile (the likely survivor is the operating system's Metal binary cache); on the Studio engine's shipped kernel a measured 33.6 s cold compile became 0.7 s warm. An un-busted "cold" number is a warm number wearing the wrong label. Same machine; recorded , cross-launch survival observed .

Noise floor (duplicate-arm floor) — a control run with the identical configuration in every timing slot, measured on the same machine the same day as the experiment it accompanies. Any effect inside the floor's band is ruled a null, not a small effect.

Failure modes that produced fake results

Every entry in this section produced a plausible-looking number before it was caught — which is the argument for the protocol, so the entries are kept on record with their numbers.

The one-time bias correction. The first attempt at handling slot bias measured it once and subtracted it as a constant from every arm. The "correction" inflated every arm by +2.58% and reported a branch that was inlined into the kernel but never executed as 6% faster. Clause 6 exists because of this run: a bias estimated once under the noise you are trying to remove is itself noise.

The in-place content A/B. An A/B that mutates one engine — remove the content under test, add the replacement, measure, swap back — measures the mutation, not the content: the record allocator fragments differently on each re-add, the scene's acceleration structure rebuilds between arms, and the machine drifts thermally across the run. On the Studio engine's starter-scene text-conversion study, the effect of interest read +16.04% on one run and +3.10% on the next, and the same probe reported a −6.87% "gain" from deleting an unused primitive from the shader module — an outcome that cannot physically happen, and the tell that the harness was being measured. The recorded rule: a content A/B that mutates one engine is measuring the mutation; build two engines from two documents instead. Starter-scene text-conversion study, .

Three harness bugs on one probe, each reading as a result. On the same study: one arm silently booted without its content — an asset path resolved against the wrong base, the loader warned and dropped it, and the arm posted a suspiciously fast time while every other assertion stayed green. Another arm read its material off a group node, which carries none, and rendered white against the opposing arm's opaque red, lighting roughly 30% more pixels. And the authoring engine was still animating between the two scene exports, freezing an orbiting lamp at different phases: mean image brightness 14.5 versus 34.1, a 2.4× lighting difference a frame-time A/B would have charged silently to the content. An image-overlap assertion caught the first two; an empty-room control — the same scenes with the subject removed, required to time identically — caught the third. The recorded conclusion: the image assertion on a timing probe is not decoration; it was the only thing that noticed.

Keep the graveyard

Negative results are first-class records: every measured null, refutation, and reverted build stays on permanent record with its numbers and rig, so dead ideas are not re-derived from intuition. Each experiment lives as a committed probe file whose comment header is the permanent primary source — purpose, numbers bound to their scene, pose, and resolution, the discipline that governed the run, and any negative outcome. Roughly 100 such probe files are committed in the Studio engine's repository, excluded from the automated test suite and run manually. Counted .

Three practices make the graveyard trustworthy rather than ornamental:

The graveyard is ultimately a credibility mechanism. A record that carries its own refuted claims — the 0.1% bias figure, the impossible −6.87% gain, the branch that was "6% faster" while never executing — is one whose surviving numbers mean something.

Related reading