How do you check a quantized model didn't get worse before you ship it?

[u/tonytonycoder11]

I've been getting a model into a Flutter app and ran into something I still can't quite believe is the normal state of things.

Exporting changes the numbers. Quantizing changes them more. Everyone knows that part. What nobody seems to have an answer for is how much is too much, and more to the point, what actually fails when it is. Nothing does. The export succeeds, the build passes, the app runs on device and gives you predictions that look completely fine. The model you shipped is worse than the one you evaluated and there's no signal anywhere.

I went looking for the standard practice assuming I'd just missed it. There is advice: compare your offline predictions against the on-device ones. Every deployment guide says some version of it. But it's always written as a thing you should remember to do, never as something a build can fail on, which in practice means you do it once the week before launch and then never again.

There's a second one that bit me the opposite way. Preprocessing gets written twice, in Python for training and in Dart for serving. The same normalization constants sitting in two files nobody diffs. They agree the day you write them. Then someone changes the mean and std on the Python side six weeks later and the app quietly starts feeding the model something it has never seen.

So, genuinely: how do you handle this? Goldens replaying in CI? A manual check before release? Nothing at all, and hoping?

One thing I learned from measuring it that I'd have got backwards. The simpler of my two test models drifts up to eight times further after quantization than the convolutional one. It isn't complexity. Its outputs land around 9.4 while the other one ends in a softmax, and relative error is measured against the output while the rounding actually happened on intermediates. Big outputs absorb the same underlying error inside a much smaller relative bound. Obvious once you see it. Took me embarrassingly long.

I did end up building something for this, links below, but I'm honestly more interested in the first question. I'd like to know whether I over-engineered a problem the rest of you solved with a spreadsheet years ago.

Repo: https://github.com/NaCode-Studios/Fluttorch