Qwen2.5-0.5B, running in your browser
No PyTorch, no transformers, no tokenizer library. The safetensors reader,
the bfloat16 conversion, the BPE tokenizer, the KV cache and the WebGPU kernels are all in
this repository. The weights
stream straight from Hugging Face; nothing is served from here but the code.
Before you press start, the honest cost: this downloads
988 MB of weights and uploads them to your GPU. On a fast connection that is a
minute or two, and it needs roughly 1 GB of free GPU-accessible memory. Nothing is
fetched until you ask for it.
It needs WebGPU with the shader-f16 feature.
Chrome and Edge 121+ on desktop have it; Safari and Firefox mostly do not yet.
What this is
A transformer inference engine written from scratch, with every correctness claim
checked against an independent implementation rather than asserted. The engine agrees with
ONNX Runtime per layer to 6.4e-5 relative and picks the same next token on every test
prompt; incremental decoding with the KV cache is bit-identical to full recomputation; and
it is slower than ONNX Runtime, by 1.3x on decode and 5.6x on prefill, which the
README
measures and explains.