When you port a model between frameworks, "does it work?" is the wrong question — the ported model will almost always work in the sense of running and producing plausible outputs. The right question is "is it the same model?", and answering that requires machinery: a numerical-parity harness. This post describes the methodology we use, which any team can build with an afternoon of setup and some discipline.
What a parity harness is
A parity harness runs identical inputs through the original model and the port, compares outputs at defined tolerances, and reports agreement statistics — as a reproducible, versioned artifact, not a notebook someone ran once. It has four components: an input corpus, a comparison contract, an execution protocol, and a divergence-localization mode.
The input corpus
A batch of random tensors is better than nothing and much worse than you think. Random inputs concentrate in the middle of the input distribution, and ports fail at the edges. A useful corpus includes:
- Representative production inputs — a few hundred to a few thousand real samples, covering the input categories the model actually sees.
- Distribution edges — the largest and smallest input sizes the service accepts, extreme aspect ratios for vision models, maximum-length and near-empty sequences for NLP models.
- Decision-boundary samples — inputs where the original model is least confident. These amplify small numerical differences into visible disagreements, which is exactly what you want from a test. Mine them by running the original over production data and keeping the samples with the smallest logit margins.
- Degenerate inputs — all-zero tensors, constant images, whatever malformed-but-accepted input your serving layer lets through. Ports and originals often disagree most on inputs neither was really trained for.
Freeze the corpus, version it, and store it next to the harness. A parity result is only reproducible if its inputs are.
The comparison contract
Decide what you compare and how close is close enough — in writing, before you run anything.
Compare at two levels. Raw outputs (logits or regression values) at a numerical tolerance, and decisions (argmax class, thresholded score, top-k set) at an agreement rate. Both matter: logit comparison catches drift that decisions mask, and decision comparison tells you what production impact the drift actually has.
Use relative-plus-absolute tolerance. A fixed absolute epsilon fails on outputs spanning magnitudes; use the standard |a - b| <= atol + rtol * |b| form. For float32 inference across frameworks, a starting point of rtol=1e-4, atol=1e-5 on logits is realistic; bitwise equality is not achievable and chasing it wastes time.
Compare distributions, not just means. Report the maximum deviation and the full histogram of deviations, not the average. Parity failures are usually a small tail of badly divergent samples hiding under a comfortable mean — and that tail is where your incident will come from.
Set a decision-agreement bar with denominators. "99.9% argmax agreement over 5,000 samples, all disagreements within the lowest-margin decile" is a contract. "Looked the same" is not.
The execution protocol
Environmental nondeterminism can drown the signal you're measuring:
- Pin both sides to evaluation mode (dropout off, BatchNorm in inference mode) — obvious, and still the most common harness bug.
- Disable TF32 during comparison on Ampere-and-newer GPUs (PyTorch enables it by default for matmuls); re-enable it later and measure its effect as a separate, deliberate decision.
- Pin cuDNN to deterministic algorithms during comparison, for the same reason.
- Run a CPU float64 pass as a reference arbiter. When GPU results disagree, the double-precision CPU run tells you which side drifted — and sometimes reveals both drifted from the mathematically correct answer.
- Start comparison from raw inputs, through preprocessing, not from pre-made tensors. Preprocessing differences are parity failures too; a harness that skips them certifies half a system.
Localizing divergence
When end-to-end parity fails, layer-wise comparison turns a mystery into a search. Register forward hooks on the PyTorch side and capture intermediate outputs on the MXNet side, then compare activations layer by layer until the first significant deviation. The first divergent layer is usually the guilty one — in our experience most often a BatchNorm parameter, a padding behavior, or a flatten-ordering mismatch, in that order. Amplification matters: a tiny early divergence that grows through the network is architectural; a late clean break is a mapping bug.
When parity fails persistently
Sometimes the harness keeps failing and each fix uncovers another divergence — commonly with custom operators or architectures ported through multiple frameworks over their lifetime. The harness then earns its keep a second way: it tells you how far the port is from faithful, which is the evidence you need to switch strategies from conversion to retraining, validated against production metrics instead of parity. Making that call early, with numbers, is far cheaper than a month of archaeology.
Keep it after the port
The harness's last job: fold it into CI. The frozen original never changes, so it remains a stable reference — every future change to the ported model (dependency upgrades, hardware moves, quantization) can be re-verified against it for free. A parity harness isn't porting overhead. It's the acceptance test, the debugger, and the regression suite in one artifact.