Predicting on-board latency before writing code

A block-level latency model, calibrated on the board, that predicted an enlarged occupancy model within 0.7%. It ranked six deployment options before any code was written and led to a backbone shared by five perception models.

QCraft 2025 – 2026 Latency modeling, deployment planning and the OneModel phase-2 design

0.7%error, 19.38 ms predicted vs 19.25 ms measured
6deployment options ranked by frame-budget share
5–9 msestimated saving from a shared backbone
  • Latency modeling
  • Profiling
  • BEV occupancy
  • Multi-task backbone
  • Deployment planning
On this page

The question

Product asked for two upgrades to the occupancy network at once: a much larger output range (from 28.8 m × 86.4 m to about 48 m × 136 m) and a frame rate doubled from 5 Hz to 10 Hz. Either one alone roughly doubles the occupancy model's cost. The chip also runs detection/tracking and map models, so the real question was:

Can both upgrades ship together? If not, what has to change elsewhere to make room?

Answering by implementing and measuring each variant would take weeks. I answered it with a model instead.

Step 1: profile by block, on the board

The network was split into blocks, and each block's latency was measured on the board, not only read from the compiler's estimate. Board timings differed from the compiler profile (about 8.8 ms profiled vs 10.0 ms measured for the whole model), and the gap was not uniform across blocks.

Step 2: decide how each block scales

Keeping the 0.3 m output resolution, the output grid grows from 96 × 288 to 160 × 448 cells, about 2.6×. Each block was then classified by how its cost depends on that grid:

Block Depends on output grid? Board measurement Estimate for larger range
Image backbone + neck (5 cameras) No: works in image space 3.92 ms 3.92 ms
View transform (image → BEV) Yes, proportional 0.92 ms 2.39 ms
Point-cloud branch Partly (neck scales) 0.48 ms 1.33 ms
Temporal fusion Yes 0.69 ms 1.79 ms
BEV encoder (backbone + neck) Yes 1.88 ms 4.90 ms
Near-field high-resolution head No: fixed region 0.45 ms 0.45 ms
Main occupancy head Yes 1.77 ms 4.61 ms
Whole model ≈ 10.0 ms 19.38 ms
structural latency model: predicted vs measuredON-BOARD LATENCY PER MODULE WHEN THE OUTPUT GRID GROWS 2.6× (MILLISECONDS)Image backbone + neck3.923.92does not scaleView transform0.922.39LiDAR branch0.481.33Temporal fusion0.691.79BEV encoder1.884.90Near-range head0.450.45does not scaleMain occupancy head1.774.61measured on board (96×288 grid)predicted (160×448 grid)The two marked modules do not scale with the grid, so their predictionequals the measurement.Whole model19.38 mspredicted19.25 msmeasured on board0.7% errorThe decision it enabledA wider range plus 10 Hz would not fit, knownbefore any code was written.The image encoder is the one large module thatdoes not scale with range, so it is the part toshare.Six deployment options ranked by frame-budgetshare: 40.1% → 54.5% → 42.3% with sharing.
Light bars are measured on the board, dark bars are predicted; the two marked modules do not scale with the output grid.

Step 3: rank the deployment options

With per-block costs known, combinations of models, frame rates and sharing could be priced as frame-budget share (at 10 Hz) without building any of them:

Option Change Share of 10 Hz budget
1 Today: three models deployed separately 40.1%
2 + larger occupancy range 44.8%
3 + occupancy at 10 Hz 54.5%
4 + occupancy reuses the image backbone and neck 50.6%
5 + map model reuses them too 46.2%
6 + map model at 3.3 Hz 42.3%

The conclusion was clear before any code: a larger range plus 10 Hz only fits if the image backbone and neck are shared. Deployment becomes two-stage. One backbone model produces feature maps, and the detection, map and occupancy heads consume them. Lowering the map model's frame rate gives further headroom if needed.

Step 4: OneModel, one backbone for five perception models

This led to the OneModel phase-2 plan I wrote: a single image backbone and neck shared by five perception models across 9 cameras at 512 × 1024. The goal was to buy latency now and capacity later.

OneModel: one shared image encoder for five perception tasksin progress: joint training runs, metrics still below releaseFIVE PERCEPTION TASKS AND 9 CAMERAS SHARE ONE IMAGE BACKBONE AND NECKToday: every task encodes on its ownbackbone + neckDetection + trackingbackbone + neckVector mapbackbone + neckOccupancybackbone + neckFront-view perceptionbackbone + neckTraffic lightsOneModel: shared up to the neck, task heads read from it9 cameras512×1024sharedbackbone+ neckDetection + trackingVector mapOccupancyFront-view perceptionTraffic lights5–9 msestimated saving perframe(rate-weighted)NOT FROM SCRATCH: RELEASED MODELS SIZE THE BACKBONE AND SEED ITS WEIGHTS1Extractfeatures from the five tasks'float checkpoints, stage bystage2PCA redundancyshrink redundantdimensions, keep theinformation-dense ones3Align channelsalign each checkpoint'schannels to a common basis4Closed-form solvea linear solve yields theneck and per-taskprojections5Check errorverify reconstruction erroron held-out clipsTwo quantization-friendly fixes ride along: ReLU6 with a surrogate gradient (ReLU6 forward, a smooth surrogate's derivative backward) and a correctedreceptive-field center offset.

The full plan (how the backbone was sized, how its pretraining weights were synthesized, how multi-task training is organized) is on the OneModel page.

  • Latency budget. Sharing saves an estimated 5–9 ms per frame. That saving is the budget for a larger backbone, so the plan first evaluated a backbone of about twice the capacity.
  • Sizing the backbone from existing models. Features from each task's released float checkpoint were extracted, and PCA measured how redundant each stage's channels were. Redundant dimensions could shrink; information-dense ones had to stay.
  • Initial weights without retraining from scratch. Channels were aligned across the existing checkpoints, PCA selected the dimensions to keep, and closed-form linear solves produced the neck and per-task projections. Reconstruction error was checked on held-out clips.
  • Training. Each step trains on one task's data across all GPUs, rotating through the tasks by planned exposure. This reuses each task's data loaders and keeps distributed training consistent.
  • Rollout. First adapt each model to the shared architecture and release it on its own. Then merge pairs, starting with tracking + occupancy. Then merge all five. A merge ships only if no task regresses.

Lessons

  • Model before you build. A block-level model answered a question that would otherwise have taken weeks of implementation.
  • Measure on the device. Compiler profiles and board timings disagree in exactly the places that matter.
  • Find the invariant. The part whose cost does not change with the requirement (the image encoder) was the part worth sharing.