Ex 2.2.7: a pilot of the syntax embeddings¶

Observations¶

• Where the component works. On the stored recipe-short checkpoints, stripping it from the readout side costs the = prediction on the red lines (1.00 to 0.85; 0.16 on t00) and nothing else. Stripping it from the embedding side costs the answer instead (1.00 to 0.55 on the red lines, 0.82 on the non-red). But turning the embeddings by the same angle in a random direction costs at least as much (0.41), so on that side the blocks seem to have learned where the = embedding sits. The pilot arms repeat the pattern. On untied, stripping the readout side costs the = prediction (0.86) and stripping the embedding side costs little (0.95 on the answer). On syntax-off-axis and untied-line, no strip changes anything (figures).
• The embedding-component table. The = embedding is at 0.26 on recipe-short and 0.93 on t00; the op words are at 0.13 and 0.34. Anchoring the blocks only leaves the = embedding at 0.25. Untying the readout brings the = embedding down to 0.12, and puts 0.31 on the readout vector. Under the whole-line labeller the ⏎ embedding takes the axis as well (0.18 on blocks-only-line, 0.28 on the readout of untied-line). syntax-off-axis holds every syntax embedding at zero by construction. The red color embeddings themselves sit at 0.81 on the reference, 0.79 on syntax-off-axis and 0.94 on untied, but at 0.49 on blocks-only and 0.47 on blocks-only-line, where nothing pulls the embedding (figure).
• Task cost. Held-out exact match is at least 0.998 on every op of every arm (table).
• Placement. m_line runs 0.335–0.400 across the pilot arms, against 0.392 ±0.020 on recipe-short; blocks-only is the low end, and one seed of blocks-only-line did not place (m_line 0.12, against 0.35 for its next seed). ᾱ at op1, the containment read, is 0.13 on untied and 0.23 on untied-line against 0.08 on the reference, and retention on untied-line is 0.93 ±0.06 (figure).
• Suppression and selectivity. Under the full-position projection the non-red mix deficit is 0.026 ±0.035 on recipe-short, where the operand-only edit gives 0.007. On the other arms it is 0.015 ±0.023 on untied, 0.011 ±0.024 on syntax-off-axis and 0.017 ±0.017 on blocks-only. Red-line mix accuracy under the projection is 0.01 on the reference, 0.02 on untied and 0.02 on syntax-off-axis. On blocks-only it is 0.17, with one seed at 0.90, and across the six ops the two blocks-only arms sit at 0.15–0.44 where the reference sits at 0.01–0.18. Under the whole-line labeller the non-red cost has a tail: three of nine untied-line seeds and one of nine blocks-only-line seeds lose more than 0.07 of the non-red mix lines under the projection, against one of the twenty reference seeds, and untied-line pays 0.019 ±0.032 under the operand-only edit as well (figure).

Why, and what we ran¶

Ex-2.2.2's E8 found the anchor axis on the embeddings of the op words and =, at a few tenths in every anchored model, and ex-2.2.3 found more of it at the heavier adopted point. That component is what a full-position projection pays for on the non-red lines, and the operand-only edit routes around it only because this grammar tells us where the operands are. In M3 there is no operand position, so the M3-shaped operator is full-position and needs the syntax embeddings clean. The tied-readout item and the design's Prep C each name a mechanism; this pilot runs both, with a third beside them.

nGPT ties the readout to the embedding, so one vector does two jobs: it is the residual stream's starting state when its token is read, and it is the logit of its token at every position that predicts it. Three mechanisms could put the axis on it.

1. The direct pull. The anchor term acts at every residual slice, the embedding included, and the labelled span covers the op word and =. Their slice-0 states are the embeddings themselves.
2. The tied readout. After a red operand the stream sits near e₁; the cheapest way to raise the logit of the token that follows is to lean that token's embedding the same way.
3. The blocks. A block that reads the component from the stream at a syntax position has a reason to keep it there, whichever table put it in.

Part A prices the component without training anything. On ex-2.2.3's stored checkpoints (recipe-short, 20 seeds, t00, 5 seeds, control-short, 5 seeds) we strip the axis component from the syntax embeddings on the input side only (the embedding, with the original table kept as the readout), the output side only, or both, and read the next-token accuracy at each position on the red and non-red probe lines, and the full-position projection's cost on the stripped model. A control turns the same embeddings by the same angle along a random direction off the axis, to read how much of an input-side cost is the turn itself. Whichever side loses the syntax predictions is where the component works.

Part B retrains ex-2.2.3's recipe-short (λ = 0.1, 50 epochs) three ways, each removing one mechanism, and reads Part A's table, E8's embedding-component table, and ex-2.2.3's placement and suppression statistics on every run. The first two also run under ex-2.2.6's whole-line labeller, which the handover proposes to adopt and which pulls the syntax positions by design.

syntax-off-axis (stored as rows-clean) is the simplest fix that keeps the table shared: after every optimizer step, the same projection that keeps nGPT's embeddings at unit length also zeroes the axis component of every non-color embedding. untied gives the readout a table of its own, initialised as a copy of the embedding, so the anchor and anti-subspace terms keep acting on the embedding and the logits are free. blocks-only is Prep C: both anchoring terms skip slice 0.

Where the component works¶

Part A on the stored checkpoints. Each panel is one of the three informative next-token predictions on the mix probe lines: = from op2, the answer from =, and the newline from the answer. The x axis is the strip condition. A filled marker is the red lines (both operands' redness at least 0.8), a hollow one the non-red lines; the small dots are the seeds and the larger marker their mean.

The same strips on the pilot arms. An arm whose syntax embeddings are already clean should show no difference across its strips; the untied arms split the two tables, so input and output strip different tables there.

The embedding-component table¶

Ex-2.2.2's E8, read on every arm: the axis component of each syntax embedding, with the mean absolute component over the 216 color embeddings beside it, and the mean signed component of the red color embeddings (redness at least 0.8), which is how far the red operands themselves sit along the axis before any block runs. nGPT's embeddings are unit vectors, so a component is a cosine. The untied arms carry a second table; its readout vectors are drawn hollow.

Task cost¶

Exact match on the held-out pairs of each op. The reference is production's twenty seeds; the pilot arms have nine each.

Where the pull lands¶

Ex-2.2.3's placement statistics on the mix probe lines, per arm. The line arms' m_line is under their own labeller's weighting, as in ex-2.2.6.

Suppression and selectivity¶

Ex-2.2.3's H4 statistics per arm: red-line accuracy under the full-position projection on each op (the removal read, lower is more complete), and the non-red mix deficit under projection and under the operand-only edit (the selectivity read). The question for each fix is whether the full-position deficit comes down to the operand-only one.

What we make of it¶

Which mechanism. Each of the three arms removed one candidate, and two of the three results point the same way. Leaving the embedding out of the anchor did not clean the syntax embeddings, so the direct pull at slice 0 was not the cause. Giving the readout a table of its own moved the component onto that table, at about the size it has on the reference, and left the syntax embeddings mostly clean.

So the logit path is what places it. Part A says the same from the stored checkpoints: after a red op2, the = prediction leans on the component on the readout side, and that is the only thing the readout side does. The untied arm shows it from the other side: stripping the readout table costs the same prediction, and stripping the embedding costs almost nothing. This is the mechanism as written in the tied-readout item.

The embedding-side strips show something else: the blocks learn to read the embeddings where the logit path leaves them. The random-direction control costs as much, and on t00, where the = embedding sits about 70° off a clean one, a stripped model is a different model. So the component is doing work, and a stronger anti-subspace term on those embeddings would pull against the logits.

Prep C. Of the three hypotheses in the design, (a) holds in the blocks: the contrast and the similarity grading are at the values of the reference, with m_line a little lower. (b) holds, since the color embeddings still lead the pull at slice 0 even though nothing pulls them there. (c) does not hold, because the syntax embeddings keep the axis.

The cost Prep C did not anticipate is completeness. On both blocks-only arms the full-position projection leaves more red lines with their answer, on every op, and on one seed it leaves most of them. The embedding-component table says why: the projection removes the e₁ component of the stream at every slice and nothing else, so anything that survives it sits off the axis.

In the reference, the anchor at slice 0 and the anti-subspace term together put the redness of a red operand onto the axis before any block runs, with the red color embeddings at about 0.8 on e₁. With nothing pulling the embedding they sit at about half that, so half of the redness enters the stream off the axis.

The blocks are as well aligned as the reference's, since the contrast and the similarity grading match. They read the redness from the stream and write it onto the axis, and the projection removes what they wrote. The off-axis half is still in the residual stream at the last slice, where the readout can use it. So "the states at the later slices are aligned" describes what the blocks add, and how well that part is aligned does not decide whether the projection is complete.

What transfers. syntax-off-axis chooses its embeddings by token class: the non-color embeddings are the ones held off the axis. That is a rule about which vocabulary entries may hold the axis, the same kind of built-in position knowledge that the mellowmax pooling was adopted to avoid, and natural language has no such class. So the constraint as run here is a fix for this grammar, and what it measures is the ceiling: what a fully clean shared table buys.

The other two arms need nothing from the grammar. Untying is available on any model, as a copy of the table, and many models already ship untied. blocks-only keeps the table shared but pays in completeness, above.

A tied-table variant we did not run would hold every embedding off the axis, so the embedding has no axis component at all. It would need no token class, but the redness would then enter off the axis by construction, the completeness cost again. That question belongs with the operator pass, where the reflection and the shaped forms are also up for decision.

The fix to carry. On nine seeds, untied and syntax-off-axis match each other and the reference on task, placement and removal. Both have a lower non-red cost under the projection on most seeds, with a two-seed tail at the level of the upper seeds of the reference. The pilot proposes the untied readout for the handover, with the full-position projection read beside the operand-only edit, and syntax-off-axis as the in-grammar ceiling it should match.

The concern from the discussion in ex-2.2.3 still stands, that a method needing an untied readout is a harder sell where the tables are tied; the all-embeddings-off-axis variant above is the tied option to test if that becomes the target. Two things to watch on the untied arms: ᾱ at op1, the containment read, is higher than on the reference, and the = embedding is lower rather than at zero.

The whole-line labeller. Under it the ⏎ embedding takes the axis the same way, since it follows the newly pulled answer, and on the untied arm it lands on the readout table like the rest.

The larger finding is the tail. On both line arms a few seeds lose a large share of the non-red lines under the projection, untied-line pays under the operand-only edit as well, and the margin on untied-line drifts down over training (the retention read). One blocks-only-line seed did not place at all.

There is no plain-recipe line arm at nine seeds here, and ex-2.2.6 has two or three, so the pilot cannot say whether the tail comes from the labeller alone or from pairing it with a fix. The handover should read the selectivity of the line labeller at more seeds before adopting it.

What it changes. The handover can adopt the untied readout, and use the full-position projection alongside the operand-only edit as its removal operator, which is what the M3-shaped operator needs. The operator pass in the D2.2 design chooses between plain projection and the shaped forms on that footing, and the whole-line labeller goes in with a selectivity check rather than by default.

Method notes¶

• Part A's strips edit the tables of a loaded checkpoint and nothing else; input and output give the tied model a readout table of its own for the scoring pass (NGPT.with_tables), so the two sides can differ. The stripped embeddings are re-normalized to unit length, as nGPT keeps them.
• The control strip replaces the axis component of each syntax embedding with one of the same size along a random unit direction orthogonal to e₁ (one fixed draw). The embedding then turns by about the same angle as under input and ends up off the axis as well.
• Part B's arms share ex-2.2.3's corpus, eval sets and probe lines (the same seeds through its prepare_corpus); the line arms train against ex-2.2.6's line-keyed probe table.
• syntax-off-axis (stored as rows-clean) applies sca.anchoring.clean_embedding_rows after every optimizer step, to every non-color embedding but the pad; blocks-only passes anchor_slices=(1, 2, 3, 4) to train_anchored, so both the anchor and the anti-subspace term skip the embedding; untied sets tie_embeddings=False on the model config, and its checkpoints carry the second table.
• The eval and score tasks are ex-2.2.3's, unchanged; on the untied arms the eval contract reads logits through the readout table and ablate_weights projects both tables.
• This is a pilot: five arms with nine seeds each (the first run had three, or two on the line arms; the rest were added on request, and the earlier seeds are memoized), Part A on five seeds of each frozen stored arm and on all twenty of the reference, and production's twenty for the reference elsewhere. Nothing here is gated, and nothing in it should be quoted as a result.