A quantization-deployment platform: one truth, one contract, block by default
Once the quantization framework worked, what remained was precision chosen by hand, errors that were hard to locate, and training that did not line up with deployment. This is the target architecture I led: the fixed-point model as the single source of truth, versioned contracts between training, conversion and the vehicle, and a blocking gate at every hop. It is a design draft.
On this page
Why redesign
By 2026 the quantization framework let model owners quantize and deploy new models themselves, without the quantization team supporting each one. Three problems remained that process alone could not fix:
- Precision is chosen by hand. Which layers get INT16 is written into model code, from experience, with no record of why.
- Error sources are hard to locate. When something mismatches, someone compares the pipeline stage by stage by hand.
- Training does not line up with deployment. The metric seen in training comes from fake-quant; the vehicle runs fixed-point arithmetic.
They share one root cause: the same thing is written once in training, once in conversion and once on the vehicle, and kept in sync by convention. Each piece works on its own. Together, a change in one place has consequences that are hard to predict.
Target architecture: one pipeline
Six design principles, each visible in the figure:
- One truth. Fixed-point semantics are the only basis for evaluation, gates and simulation. Fake-quant is a training proxy, and its metric is not the release metric.
- One contract. Codebases exchange only versioned contract artifacts, never file-name conventions or verbal agreements.
- One kernel, differences outside. There is one flow. Model differences go into adapters, chip differences into capability files, precision into a precision profile. Whatever can be data is not a branch.
- One implementation. IR export and rebuild are written once and shared by training, conversion and simulation.
- Block by default. Every hop has a gate. Over the limit fails and reports the first divergence; a waiver must be written into the threshold file with an expiry date.
- Environment as code. Toolchain versions, images and configs are checked in and recorded in every artifact's identity.
Seven contracts
Contracts turn convention into files that can be validated. Each carries a schema version.
| Contract | Content | Written by | Read by |
|---|---|---|---|
| Identity manifest | Source, environment, quantization state, graph hash, toolchain, artifact hashes, parity and performance | Training writes the first part, conversion appends, release seals it | Every stage |
| IR metadata | Schema version, quantization scheme, rounding rule, target chip, graph hash | Export | Every consumer; a version mismatch is rejected |
| IO contract | Name, order, shape, dtype, scale and meaning of every input and output, plus pre- and post-processing | Model adapter | Conversion, on-vehicle parameter generation |
| Golden IO | A few frames of real deployment-domain input, with reference outputs at every hop | Export | Parity engine, board spot checks, on-vehicle consistency tests |
| Precision profile | Default precision plus overrides by module name, each with evidence | Precision planning | Quantization kernel |
| Target capability | Supported dtypes, rounding, unsupported operators, INT16 fallback rules | Chip owner | Precision legalization, build routing |
| Thresholds | Error limits per hop and per output, and waivers with expiry dates | Model owner and quantization owner | Parity engine |
Hop gates: one truth, two zones
| Gate | Compares | Criterion |
|---|---|---|
| G1 training graph → IR | Quantization-state fingerprint before and after export | Identical |
| G2 IR → rebuilt model | All scales and dtypes; outputs under a fixed evaluation order | Scales identical |
| G3 rebuilt model → QAT HBIR | Integer-code error per output | Within the limit |
| G4 QAT HBIR → fixed-point model | Error introduced by going fixed-point | Per-output budget |
| G5 fixed-point model → HBM | Output bytes | Byte-exact |
| G6 HBM → board | Output bytes | Byte-exact, sampled at release |
| G7 fixed-point model → fixed-point simulator | Output bytes | Byte-exact |
| G8 simulator ↔ vehicle | Final outputs for the same raw frame | Consistent |
| M release metric | Fake-quant metric vs fixed-point metric | Gap within budget |
Most of the comparison tools exist today. What the design adds is turning “record” into “block”, and reporting the first node over the limit automatically when a gate fails.
Precision planning loop
Per-layer precision is tuned by hand inside model code today. The design replaces that with a process that has inputs, evidence and can be re-run. It rejects the combinations the target chip cannot run, runs sensitivity experiments per module group, brings in the on-board latency cost table, and chooses greedily by metric gained per millisecond within the latency budget. The output is a precision profile with a reason for every override: why it was raised and what it costs in latency. Model code no longer holds precision settings.
Quantize once, derive for every chip
Quantization training is done once, and every chip derives from the shared weights and precision intent. Chip differences may appear in three places only, all of them data: the capability file, the legalized precision plan, and a declarative graph-rewrite config. There are four migration levels, from direct export to a separate QAT. The gap between those two is measured first, then each chip gets its default level. Adding a new chip becomes one capability file, one build back end and one on-vehicle runtime back end.
Rules that do not rely on discipline
- Every dependency rule maps to a check that can fail. The quantization kernel does not import concrete models; exactly one directory may import the chip vendor's quantization library; chip names appear only in chip directories and capability files; per-layer precision comes only from the precision profile.
- Seven test tiers, slower and closer to the device as they go up. Unit tests (seconds, no GPU) → contract tests (every merge) → GPU smoke → cross-environment alignment → build and simulation regression (daily) → board spot checks → release gates.
- Legacy only shrinks. Hand-written precision, chip branches and environment switches each have an allow-list, and CI only lets the lists get shorter.
Migration approach
Measurement and gates come first, code migration second: first measure and block, then choose, and only then release through the new pipeline. A pilot model goes first and models outside the pilot keep the current flow throughout. Every step is accepted on “behavior unchanged”.
Notes from the design phase
- Measure before changing. Once “it feels messy” became specific numbers, the priorities sorted themselves.
- Start from what exists. Taking stock of what each gate already has and what it lacks keeps the design from becoming a rewrite.
- Turn conventions into files. Anything that rests on a verbal agreement will fail quietly in some toolchain upgrade.