Perplexity under each weight scheme, plus whether the model still picks the same next token as the fp32 ONNX Runtime oracle. Weights are quantized and dequantized on the way to the GPU, so the kernels are unchanged and the only variable is the weights.
text a fixed passage, held in this page, tokenized with the model's own tokenizer
window every token after the first is predicted from all tokens before it,
with no sliding window and no stride, so each prediction sees its
full left context and no token is scored twice
scored every position except the first, which has nothing to condition on
metric exp(mean negative log-likelihood of the actual next token), natural log
decoding none: this reads the model's distribution, it does not sample
identical the same token sequence and the same positions for every scheme
Stating this first is the point. Perplexity is easy to move by changing the stride, the window, or which positions get counted, so the protocol is fixed in advance and every scheme is scored on exactly the same positions.