Transformer blocks, residual paths, and position

After token embedding, a decoder-only transformer repeatedly updates a hidden vector at every position. Each block typically includes an attention sublayer and a position-wise feed-forward sublayer, combined with residual connections and normalization. The final hidden states are transformed into vocabulary logits. Exact arrangements vary, so inspect the architecture configuration rather than assuming every named transformer uses identical ordering.

Attention mixes information across permitted positions. The feed-forward sublayer transforms each position's features using shared learned weights, often expanding to a wider intermediate representation before projecting back. With causal masking, the hidden state for a prefix position cannot depend on future tokens. That property is central to why old key and value representations can later be reused during incremental generation.

A residual update has the form x plus F(x). The sublayer learns a change to the existing representation while an identity path remains available. In backpropagation, that identity path contributes directly to the derivative, which can help optimization in deep networks. It is not a guarantee against instability or a promise that every layer preserves an interpretable fact.

Consider x = [2, -1] and a sublayer output [0.1, 0.3]. The residual result is [2.1, -0.7]. Without the addition, the output would contain only [0.1, 0.3]. The numerical example is simple, but it captures why a sublayer can make a small correction instead of reconstructing the whole representation. Learned projections and normalization can still substantially change information over many layers.

Layer normalization uses statistics across a position's feature coordinates, followed by learned scale and often shift parameters. RMS normalization instead scales using the root mean square and does not perform the same mean-centering operation. Architectures may normalize before a sublayer, after it, or use additional variants. These choices alter the computation and training behavior; the names are not interchangeable.

For x = [3,4], the root mean square is the square root of 12.5, approximately 3.536. Dividing by it produces approximately [0.849,1.131] before learned scaling. This operation changes magnitude but retains some relative coordinate structure. A small epsilon avoids division by zero. Normalization is internal feature conditioning, not a substitute for validating user data or calibrating final output probabilities.

Rotary positional embedding applies position-dependent rotations to paired query and key coordinates. For a pair [a,b] and angle theta, the rotated pair is [a cos theta - b sin theta, a sin theta + b cos theta]. Different coordinate pairs use different frequencies. Rotation preserves the pair's norm while changing its orientation, so position influences query-key dot products.

A useful identity is that the dot product of q rotated by m theta and k rotated by n theta depends on their relative rotation, proportional to n-m. Rotating both by an additional shared angle leaves their dot product unchanged. This supplies a structured relative-position effect within attention. Actual models use many coordinate pairs and architecture-specific frequency or scaling settings; the two-coordinate demonstration is only the underlying mechanism.

Modern decoder variants can use gated feed-forward activations, grouped key/value heads, tied embedding and output weights, or mixtures of experts. These choices affect parameter count, active computation, memory traffic, and cache shape. A model's total parameter count does not by itself specify per-token compute or serving memory. The next lesson focuses on one especially practical consequence: the cache retained for previous tokens.

The example verifies norm preservation and a shared-shift dot-product identity for a single rotary pair, then performs a visible residual update. It makes no claim about a particular deployed model. When inspecting an open architecture, write down layer count, hidden dimension, head layout, position settings, normalization type, and vocabulary size. These configuration facts help explain behavior more concretely than a model-family label alone.

Two query/key coordinate pairs are rotated at different positions, then both positions are shifted by seven. Their dot product remains equal within floating-point tolerance. The separate residual calculation demonstrates addition. This is a RoPE identity check, not a complete positional implementation for a pretrained model.