How LLMs Actually Work
Most transformer explainers get lost in the math. This visualizes how information flows through the network end to end.
There are two distinct information highways in the transformer architecture:
The Residual Stream (Vertical): Flows vertically through layers at each position. This is the "thought process" for a single token.
The K/V Stream (Horizontal): Flows horizontally across positions at each layer. This is how tokens "look back" at previous tokens.
The Two Information Highways
The Transformer combines vertical processing (refining each token's representation through layers) with horizontal information sharing (attending to past context via K/V). This creates an incredibly rich space for information to flow.
The Attention Computation
At each layer, for each position (token), the network performs a specific set of operations:
The incoming residual stream is used to calculate K (Key) and V (Value) vectors for that layer.
These K/V values are combined with all previous K/V values (the "context").
The attention mechanism compares the current Q (Query) with past Keys to calculate attention scores (a "heat map").
Based on these scores, it gathers a weighted sum of Values from the past.
This output is processed by an MLP (Feed Forward Network) and added back to the residual stream.
What Q, K, and V actually mean:
The K/V cache isn't just a performance trick—it's the accumulated memory of the sequence. Each position adds to the pool of available information without destroying what came before.
Information Paths from A → B
Path 1: Information travels UP through attention and MLP at position 0, then is retrieved RIGHT 2 positions via the K/V stream.
Information Paths & Superposition
Information can take an astronomical number of paths through the network. It can travel "up" (processing existing information) or "right" (incorporating past context) in any order.
Between a past state (Layer $i-1$, Position $j-2$) and a future state (Layer $i$, Position $j$), information could:
- Travel UP through layers, then be retrieved RIGHT by attention later.
- Be retrieved RIGHT by attention immediately, then travel UP through layers.
- Or any combination of steps in between.
This dense connectivity means the model isn't just "remembering" usage—it's experiencing computation as a continuous, interferometric process across time.
The Causal Graph & Foliations
Hover over a node to see its "light cone" (everything that influences it). The red diagonal lines show one possible ordering of computations.
The red diagonal lines represent "foliations"—valid orderings for when computations can happen. Unlike traditional views, computation doesn't have to proceed strictly left-to-right or bottom-to-top. Any ordering that respects causal dependencies (can't read from a node before it's computed) is valid.
This connects to Wolfram's Physics Project—the idea that causality, not strict temporal ordering, is fundamental.
Can LLMs Introspect?
Because the architecture connects every future state to every past state through these precise paths, the popular idea that "LLMs cannot introspect" is technically incorrect from an architectural standpoint. The mechanism exists to retrieve and process its own past internal states.
Whether they are trained to effectively leverage this for human-like introspection is a separate question, but the machinery is undeniably there.