Where the lean comes from¶

Spill-by-position found that the removalRemoval How much of difference the edit takes out: the net drop in its score under the edit, as a share of the way from the clean score to the target null. and the spill share the example answers, and that a syntax token latched onto e₁ shapes the spill. The τ × λ_a sweep found that the spill persists with no latchLatch A syntax token embedding on e₁: a token every context contains, so putting it on the anchor direction satisfies the pull at the embedding for every context at once. Found by spill-by-position. at all. That left two questions: what causes the spill when there is no latch, and why the anti-subspace term does nothing about it.

This report re-analyzes the stored checkpoints of ex-2.2.23 (24 anchored and 24 control runs, at 200 and 400 epochs) and the no-emb trials of the sweep near the recipe τ (11 trials), with no new training.

Observations¶

• What e₁ holds on the other ops (E1): at the example answers of other ops, e₁ holds a per-example judgement of whether that one example fits difference. The spillSpill The largest net drop in score on another op under the edit. The selectivity criterion is 0.02. does not follow it, so the spill is not the concept firing on the wrong contexts.
• Lightness on e₁ in the embedding table (E2): every anchored run loads lightness onto e₁ in the color embedding table, darker colors further along. The leanLean Lightness on e₁ in the color embedding table: darker colors sit further along the anchor direction, a stand-in for difference at the slices where nothing contextual exists. is about three times larger on runs with no latchLatch A syntax token embedding on e₁: a token every context contains, so putting it on the anchor direction satisfies the pull at the embedding for every context at once. Found by spill-by-position., and the latch decides how the run spills.
• The lean fades with depth (E3): the lean is strongest at the embedding and fades through the blocks. It is there at 200 and 400 epochs and on the no-emb trials alike, and the readout table has none of it.
• The edit by slice (E4): editing only blocks 2 to 4 removes most of difference with almost no spill. Editing only the embedding gives half the removalRemoval How much of difference the edit takes out: the net drop in its score under the edit, as a share of the way from the clean score to the target null. and all the spill.

Scope¶

This is a re-analysis with no preregistration and no gate. It covers every run of ex-2.2.23: 12 anchored runs at 400 epochs and 12 at 200, each with its control at the same model seed, scored on ex-2.2.21's held-out contexts. The prose quotes the 400-epoch runs, since they have learned every op, and the 200-epoch runs are shown beside them.

The sweep trials are the no-emb trials with τ between 0.05 and 0.3, measured on the tables and states only.

The measurements follow ex-2.2.23 and spill-by-position. The task score of an op is the expected exact matchExpected exact match The task score, as the chance that a drawn answer agrees with a true one. With drawn answers, the correct answer to a line is spread over two or more colors, so we take the chance that an answer drawn from the model agrees with one drawn from the true distribution. Its ceiling is that same chance for the true distribution against itself (Σq²), below 1 on the ops that round. of the answer on held-out contexts of that op. RemovalRemoval How much of difference the edit takes out: the net drop in its score under the edit, as a share of the way from the clean score to the target null. is the drop in the difference score under the edit, net of the same edit on the paired control run, as a share of the way from the clean score to the target nullTarget null The score an ideal predictor would get on difference contexts if it had lost that op and answered from the remaining ones.. SpillSpill The largest net drop in score on another op under the edit. The selectivity criterion is 0.02. is the largest net drop on any other op; the selectivity criterion is 0.02.

The full edit removes the whole e₁ component at every position and slice; E4 restricts it to a set of slices. The alignment α is the cosine of the state with e₁,¹1 Cosine similarity: how closely two vectors point the same way, ignoring length. 1 means the state lies along e₁, 0 means it is perpendicular to it. averaged over slices 2 to 4 where a section says so.

The checkpoints are the ones the recipe was tuned on, so the seed-to-seed spread here is the spread the recipe has, with nothing held out. The slice-restricted edits of E4 are on models trained with every slice pulled, so they say where the removable part of the concept sits in those models. Only a new run can say whether a pull kept off the early slices would put the concept in the later blocks.

What e₁ holds on the other ops (E1)¶

Is the spillSpill The largest net drop in score on another op under the edit. The selectivity criterion is 0.02. the concept itself firing on contexts that are not difference? At an example answer the anchor follows the posterior given that one example (example-evidence), so a mix context whose first example happens to fit difference should have some alignment there.

It does, strongly. On the 400-epoch runs the alignment at an example answer of another op is near 0.9 when that example fits difference on its own and near 0.1 when it does not, on every seed, latched or not. So e₁ at the example answers holds a clean per-example judgement, and other ops' contexts get it wherever an example fits.

But that is not what spills. The contexts with no fitting example lose the most under the edit, and the few contexts with two or three fitting examples gain a little, so the correlation between the drop and the number of fitting examples is negative on every run (about -0.18). The control loses a little under the same edit at every count, with no pattern.

So the spill comes from something the edit removes at every context of the affected op, whatever its examples say. The one place the pull reaches but no context can is the embedding table (E2).

At slices 2 to 4 the alignment at the query = is small on the other ops' contexts, and it rises with the number of fitting examples. That fits the query position tallying the per-example judgements, though this analysis can't tell whether it is a count or a posterior.

Lightness on e₁ in the embedding table (E2)¶

At the embedding slice the state at a position is the token embedding alone, with nothing yet read from the context. The pull asks each labeled difference context to have some position with α near 1 at that slice too, so it can only be met at the token level: by a latchLatch A syntax token embedding on e₁: a token every context contains, so putting it on the anchor direction satisfies the pull at the embedding for every context at once. Found by spill-by-position., or by a statistical stand-inStand-in Something token-level that satisfies the pull where no contextual concept can: a latch or a lean., where tokens more common in difference contexts sit further along e₁.

Lightness is such a stand-in. difference answers are the channel-wise absolute difference of the operands, so they are darker than the colors around them.

Every anchored run has this leanLean Lightness on e₁ in the color embedding table: darker colors sit further along the anchor direction, a stand-in for difference at the slices where nothing contextual exists.: the correlation between lightness and the e₁ component is about -0.7 on all 24, with the same sign on every one. On the controls it averages zero but runs from -0.5 to 0.6, because lightness is a major direction of the color table and a fixed axis picks up some of it by chance, with a sign that depends on the seed. The spread of the component on the controls (0.11) is what projecting unit vectors onto a random direction in 64 dimensions gives.

The size of the lean depends on the latch. Where a syntax token sits on e₁, the pull is met at that token and the color table barely moves, so the spread of the e₁ component over the colors is small (0.06). Where nothing latches, the colors take the whole pull, and the spread is about three times larger (0.26).

The runs latched on ⏎ barely spillSpill The largest net drop in score on another op under the edit. The selectivity criterion is 0.02., and their color table hardly leans. That token ends the context and attention stops at it, so nothing downstream reads it.

The runs latched on , have a small lean and the largest spill. The , ends every example, and on these runs nearly the whole , embedding lies along e₁. So the projection that removes e₁ removes the token itself, and every context of every op loses its example boundaries, as spill-by-position saw.

The runs with no latch have the largest lean and spill moderately, onto darken on all but one of them. The right panel puts the three groups on one line: the further the lightness correlation in the states has moved from zero, the more the run spills.

The lean fades with depth (E3)¶

If the leanLean Lightness on e₁ in the color embedding table: darker colors sit further along the anchor direction, a stand-in for difference at the slices where nothing contextual exists. is a property of the embedding table, it should be strongest at the embedding and fade as the blocks add contextual signal.

The lean is there at 200 epochs as at 400, so it comes with the pull; training longer only grows it on the runs that lose their latchLatch A syntax token embedding on e₁: a token every context contains, so putting it on the anchor direction satisfies the pull at the embedding for every context at once. Found by spill-by-position.. On the latched runs the lean in the states falls away at slice 1, which fits the latch taking the pull from the color table.

The no-emb trials lean just as far at the embedding (-0.7), although that slice is not pulled. Leaving slice 0 out of anchor_slices takes the anti term off it as well, so nothing holds the table in place while the pull at block 1 moves the embedding beneath it.

The controls spread to either side of zero at every slice, a different sign on each seed, as their embedding tables do (E2). The anchored runs all lean the same way.

The readout table has no lightness on e₁ on any run (the correlation is below 0.13 in size), so the lean is on the input side only.

The edit by slice (E4)¶

Is the spillSpill The largest net drop in score on another op under the edit. The selectivity criterion is 0.02. caused by the edit at the early slices? Here the full edit is applied to one set of slices at a time, on every anchored run.

At 400 epochs the embedding slice alone gives half the removalRemoval How much of difference the edit takes out: the net drop in its score under the edit, as a share of the way from the clean score to the target null. and all the spill: the spill under the embedding-only edit is the same size as under the full edit, run by run. Blocks 2 to 4 alone give most of the removal, with spill within the criterion on 7 of the 12 runs.

Block 4 alone does nothing. That matches the readout having no lightness on e₁, and suggests the concept is read out of the stream before the last block.

The 200-epoch runs spill little under any edit because most of them are latched on ⏎. Their removal under the late edit is lower, because the unlatched ones leanLean Lightness on e₁ in the color embedding table: darker colors sit further along the anchor direction, a stand-in for difference at the slices where nothing contextual exists. at the embedding like the 400-epoch runs.

But the late edit varies a lot between runs. On 3 of the 7 unlatched 400-epoch runs (seeds 700, 708, 710) the edit on blocks 2 to 4 removes much less (0.14, 0.60, 0.09) than block 1 alone does on the same runs (0.95, 0.87, 0.85), while on the other unlatched runs it removes as much as the full edit. On those runs the model has come to compute difference partly from the lightness that the lean put on e₁. So taking the lean away is part of how the full edit works, which makes a late-only edit on the current recipe a weak fix on its own.

Discussion¶

The spillSpill The largest net drop in score on another op under the edit. The selectivity criterion is 0.02. has one source on these runs: the pull at the slices where no contextual concept can exist. At the embedding, and largely still at block 1, the state is the token, so the pull is met by whatever token-level feature separates difference contexts from the rest. The cheapest is a syntax token shared by every context (the latchLatch A syntax token embedding on e₁: a token every context contains, so putting it on the anchor direction satisfies the pull at the embedding for every context at once. Found by spill-by-position.); failing that, the darkness of the answers (the leanLean Lightness on e₁ in the color embedding table: darker colors sit further along the anchor direction, a stand-in for difference at the slices where nothing contextual exists.).

The edit then removes the stand-inStand-in Something token-level that satisfies the pull where no contextual concept can: a latch or a lean. from every op, and the ops that depend on lightness lose accuracy. This fits the four observations and the two earlier reports, and the edit-by-slice result is the same mechanism seen from the other side.

Two things stop the anti-subspace term from keeping the color table off e₁. The first is normalization. The anti term is the mean of cos² over every live position and every slice, so each color embedding is one of a few thousand terms in a mean, and moving one of them along e₁ costs almost nothing.

The pull, by contrast, divides by the number of labeled contexts. Mellowmax at the recipe τ²2 Mellowmax: a smooth version of the maximum over positions, here of the alignment within a context. A small τ makes it favor the best position more sharply; a large τ makes it closer to the mean. also concentrates the gradient of each context on its best position, so the embedding of the darkest token in a difference context gets most of the pull of that context.

A rough count puts the pull on a color embedding an order of magnitude or more above what the anti term costs it at the hold ratio, and further above once the anti term anneals.³3 A rough count: at the hold the anti weight is about 0.03, spread over roughly 300 live positions per window and five slices, so one embedding at cos² = 0.3 costs on the order of 0.03 × 0.3 / 1500 per window. The pull on the same embedding, where it is the best position in a labeled context, is on the order of 0.1 × (1 − α) / (labeled contexts per window, about 3) per window at each slice it is pulled on. The ratio is in the tens, and it depends on how many contexts the token appears in, so only the order of magnitude is meaningful. So a lean of this size is the cheapest way the model has to satisfy the slice-0 pull, and the anti term only slows it a little.

The second is that the anti weight is set relative to the anchor weight, so the sweep that varied λ_a scaled both terms together and left the ratio between them unchanged. That is why λ_a barely mattered.

The no-emb trials fit the same account: nothing holds their table in place (E3), and the edit still touches slice 0, so the lean is still removed from every op. That is also why ex-2.2.22 found that every slice restriction spilled more, with the states of other ops further along e₁ at the first two slices. The restriction took the anti term off the slices where the stand-in lives while the edit went on removing it.

If this account is right, a pull kept off the first two slices, with those slices held off e₁ by the anti term or by a hard constraint on the embedding table, would leave the model nothing to meet the pull with but a contextual feature, and the edit would then remove difference without the lightness.

The unlatched runs where the late edit removed little show the risk: on the current recipe the concept is partly built out of the lean, and a run with no lean has to build it another way. Whether it can is a question for a new run, since the stored checkpoints only say where the concept sits once the lean is there.

Glossary¶

Latch

A syntax token embedding on e₁: a token every context contains, so putting it on the anchor direction satisfies the pull at the embedding for every context at once. Found by spill-by-position.

Lean

Lightness on e₁ in the color embedding table: darker colors sit further along the anchor direction, a stand-in for difference at the slices where nothing contextual exists.

Stand-in

Something token-level that satisfies the pull where no contextual concept can: a latch or a lean.

Removal

How much of difference the edit takes out: the net drop in its score under the edit, as a share of the way from the clean score to the target null.

Spill

The largest net drop in score on another op under the edit. The selectivity criterion is 0.02.

Target null

The score an ideal predictor would get on difference contexts if it had lost that op and answered from the remaining ones.