LLM Transformer Machinery
The durable mental model for modern LLMs is a repeated transformer block: token IDs become embeddings, position enters the representation, attention mixes token information, feed-forward layers process each position, residual/norm paths keep deep stacks trainable, and the model predicts one next token at a time.
Why It Matters
0xkato's article is useful as an onboarding map because it keeps the transformer architecture legible without turning the explanation into a derivation. The article walks through tokens, embeddings, positional encoding, attention, multi-head attention, feed-forward networks, residual streams, layer normalization, the next-token loop, and the difference between architecture and trained weights. That sequence is the right debugging order for most model-behavior questions: ask which layer of the stack owns the phenomenon before reaching for a vague "the model knows" explanation. Source: 0xkato, 2026-06-01
Machinery Map
| Layer | Operational reading |
|---|---|
| Tokenization | Text becomes token IDs; tokenizer choice changes cost, multilingual behavior, and failure modes such as character counting. |
| Embeddings | Token IDs become learned vectors; this is the first model-internal meaning surface. |
| Positional encoding | The model needs order and distance; modern open models commonly use RoPE-style relative position signals. |
| Attention | Tokens compare query/key/value projections so context can move between positions. |
| Multi-head attention | Different heads can track different relationships; KV caches and grouped-query attention make generation cheaper. |
| Feed-forward network | Each position gets independent nonlinear processing; dense FFNs carry a large share of model parameters, while MoE variants route tokens through selected experts. |
| Residual stream and normalization | Blocks add into a running representation; normalization keeps the stack stable enough to train deeply. |
| Next-token loop | Generation repeatedly converts the last-position vector into logits, samples or selects a token, appends it, and continues. |
Kevin Stack Reading
This page is a reference for Context Engineering and LLM Inference Serving Architecture, not a new tool. It clarifies why context windows are an attention budget, why long prompts create quadratic attention pressure, why KV caches matter for streaming and latency, why private page is affected by tokenization and decoding, and why post-training changes model behavior without changing the broad transformer skeleton. Source: 0xkato article review, 2026-07-03
Use it when a discussion drifts into model mysticism. Most practical questions can be grounded in one of four concrete surfaces: input representation, cross-token information flow, per-token computation, or decoding policy.
Weights as editable code
Rikiya Takehi reports early signs that frontier models can read and write neural-network weights directly. GPT-6 Astra named four game policies from raw weights alone (same architecture, inputs and actions); patched a stuck PPO car policy by editing 3.5% of its weights, from one failed trace; wrote a Breakout network's weights from the game rules that outscored PPO trained for 500M steps; and fixed Qwen3.5-0.8B's known reasoning loops by changing 0.016% of its weights with a budget of 50 forward passes, improving math up to six times with no gradients. An independent rebuild from the published logs scored 51.0% on MATH-500. The author calls it weight-space programming. Source: X/@rikiyatakehi thread, 2026-11-02; Source: blog
Read it next to Cheng Lou's Handwired on Cheng Lou (Chenglou), which goes the other way and compiles code into exact weights. Both treat weights as a program format. Treat the results as early research, not a method to rely on.
Why local decoding is memory-bound
Each generated token needs one full pass through the model, and each pass reads every weight from memory. A base M5 reads memory at about 153 GB/s and a 4-bit 9B model is about 5.2 GB, so ordinary decoding tops out near 29 tokens a second however fast the GPU is. llama.cpp (22.0 tok/s) and MLX (25.1) sit just under that ceiling. Mirai's uzu reached 92.1 by producing several tokens per pass: a small drafter proposes the next 16 positions, a 56.7M-parameter adapter turns them into a tree of coherent candidates, and the 9B model checks the whole tree in one pass and keeps the longest path it agrees with. The main model still decides every output token. Gains are largest on code and structured output and smaller on open-ended chat. Version checked: uzu 0.6.15 (2026-10-30), MIT. Source: X/@_avichawla, 2026-10-27; Source: trymirai/uzu
A step-by-step numeric walk through attention: words to vectors, then query, key and value matrices, scores, softmax and the weighted sum. Source: X/@amitiitbhu, 2026-04-03
KTransformers runs very large MoE models such as DeepSeek-R1 (671B) on a single 24 GB GPU by offloading experts to CPU memory, with 3 to 28 times faster inference than naive offload, and fine-tunes DeepSeek-V3 on four 4090s. Source: X/@thesupermanmx, 2026-07-18; Source: X/@thesupermanmx, 2026-07-18
Timeline
- 2026-07-03 | Created from deep review of 0xkato's "How LLMs Actually Work" bookmark. Promoted from a hub-only onboarding reference into a concept page because the source is substantive, article-backed, and directly useful for model/debugging explanations. Source: X/@0xkato, 2026-06-03