Ex 2.2.22: Localized by depth, various pull caps, and contexts of varying length¶

Ex-2.2.21 found that an edit applied at every position can take difference out of an anchored model gradually while the other ops stay as they were, at least when the pull is capped. This scout trains 9 new conditions at 3 seeds each, and reuses ex-2.2.21's runs as references, paired by model seed.

Findings¶

• Localized by depth (E1) — every slice restriction lets the edit spill onto other ops past the gate, on nearly every run.
• Various pull caps (E2) — the selectivity does not fall off steadily with the cap; runs past the gate turn up at every cap from 0.8 to 0.95.
• Mixed counts keep the recipe near its ceiling (H1) — partial. k-mixed falls short of hinge by a little more than the tolerance, inside the seed band.
• The anchor and the posterior (E3) — at the example answers, α rises with the posterior on every anchored condition, at a fixed answer index too.
• The edit lands on the target null (H2) — miss. The edit moves each replicate run a little under half of the way to the target null.
• Where the distance from the target null remains (E4, post hoc) — the edit removes nearly all the mass on answers only difference gives; what remains is on answers other ops give, shared out differently from the target null.
• Training trajectories (E5, post hoc) — most runs rise to a plateau and then rise again; on several conditions the run at model seed 702 never makes the second rise.
• Decision — keep every slice in the pull and three examples per context, drop the cap, and look into the seeds before confirming it.

Why this experiment¶

In the in-context grammar the model never sees the name of an op. It reads a few solved examples, works out which op must be in use, and applies that op to the query. So when we anchor difference, we ask the residual stream to lie along one direction, e₁, on contexts whose examples follow difference. Ex-2.2.21 found that the anchor settles on the example answers, where the context shows the most about its op, and that an edit removing the e₁ component at every position takes difference out. The edited model seemed to answer difference contexts much as an ideal predictor would if it no longer knew that op; H2 scores that more formally.

We now consider three changes to our recipe.

First, which slices the pull acts on. A slice is one of the depths along the residual stream where we can read the state: the token embedding, then the output of each block, the last of which feeds the readout. The current recipe pulls every slice. A condition that left the embedding slice out had the best task score of any, but its edit spilled onto other ops. An op is inferred from the whole context, so it seems likely to live in the middle of the stack, which suggests leaving out the last slice too. But leaving slices out changes the anchor weight in effect as well, since the term averages over the slices it pulls, so each new restriction comes with a condition at the weight that keeps the effective pull as it was.

Second, how hard the pull is on nearly aligned states. The hinge condition stops pulling once a state reaches an alignment of 0.8 with e₁, and its edit stayed within the selectivity gate; the uncapped condition just missed it. The cap of 0.8 was arbitrary, so two caps between 0.8 and no cap ask whether the selectivity falls off gradually.

Third, how many examples a context has. Recent models trained on three examples per context. We would like to be able to test whether the anchor grades with the evidence, and given a whole context of three examples, the posterior on difference takes only a few distinct values. Varying the count from one context to the next gives it more, and training on that lets the model learn that contexts vary in length. That changes the recipe, so it has to show it still gets near its ceiling.

Parameters¶

Every new condition changes one setting from the hinge condition of ex-2.2.21, our best recipe so far. The base recipe is: the seven-op set; replacement op noise of ρ = 0.3 (the chance that an example follows some other op); the newline mask; 200 epochs; d64-L4; and the whole-line label. Like the hinge condition, no condition has verification lines (examples whose answer the model judges as right or wrong); ex-2.2.21 found they leave completion unchanged.

The slice conditions. The slices are numbered from 0, the embedding, to 4, the output of the last block. no-emb leaves out slice 0, no-last slice 4 (the input to the readout), and middle both, so only the three slices between them are pulled. The anchor term and the anti-subspace term both average over the slices they act on, so at a fixed weight, pulling fewer slices pulls each one harder: by a quarter for no-emb and no-last, and by two thirds for middle. Each -matched twin scales the weight down by that factor. So the plain condition and its matched twin bracket the two readings of a restriction: a change in where the anchor is, and a change in how strongly each slice is pulled. Leaving a slice out also drops the anti-subspace term there, so the states of other ops at that slice are free to sit on e₁. No condition separates that from the anchor term, so E1 compares the two terms left out together.

The cap conditions. The pull capped at 0.9 and at 0.95. With the hinge condition at 0.8 and ex-2.2.21's whole-line condition uncapped, that makes four levels.

The count condition. k-mixed trains on a corpus where each context draws its number of examples, k, uniformly from 1, 2, 3, 4, 5. The mean is three, as in the fixed corpus, so an epoch has about as many tokens and steps. The held-out set has every count. The figure below shows the posterior on difference given a whole context, at each count and pooled.

Each count gives a few steps of its own. One example leaves most contexts in some doubt, and puts about a quarter near zero, where the one example fits another op better. Two examples is the least decisive count: about half the contexts sit near 0.4, where the two examples fit some other op about as well. From three examples on, more and more contexts are near-certain. Pooled, the curve has the steps of every count, so it rises in many small steps. The share in the "middle band" (a posterior from 0.5 to 0.95, where the examples favour difference and still leave some doubt) is lower pooled than at three examples (31% against 42%), so what the mix adds is more distinct levels.

Drawing the counts uniformly keeps the mean at three, though, as above, it puts fewer contexts in the middle band than three examples alone. One example contributes the most contexts in doubt, and pooled over the counts, the share of difference contexts whose examples favour some other op is about what it is at three examples, so the short contexts do not add much ambiguity to the label overall. H1 reports the skill at each count, which would show whether the short contexts are too hard to learn from.

The seeds. The model seeds of ex-2.2.21's three-seed conditions, 700 to 702, so every comparison is paired by seed. In ex-2.2.21 the last of these took a slow path through training on most anchored conditions, though not on no-emb, so a setting can change it; the pairing shows whether any here do.

The replicate. Three more runs of the hinge condition, at seeds 703, 704, 705, which it was not trained at in ex-2.2.21. H2 is scored on these (that section says why). With them, 30 runs in all.

Measurements¶

The measurements are the same as in ex-2.2.21, with two additions.

Task score. Expected exact matchExpected exact match The task score, as the chance that a drawn answer agrees with a true one. With drawn answers, the correct answer to a line is spread over two or more colors, so we take the chance that an answer drawn from the model agrees with one drawn from the true distribution. Its ceiling is that same chance for the true distribution against itself (Σq²), below 1 on the ops that round. (EEM) on held-out contexts: the probability the model puts on the right answer under the true op. Each run is compared with the control, the unanchored condition of ex-2.2.21, at the same seed. For k-mixed it is also reported at each count, beside the Bayes ceiling at that count (the score of an ideal predictor that weighs every op by how well it fits the examples).

Where the anchor sits. The alignment α of the state with e₁ (their cosine) by role and slice, on difference contexts and on the others, and the op margin: how far difference contexts sit along e₁ beyond the rest. Ex-2.2.21 measured the margin at the last slice, which no-last and middle leave unpulled, so here it is reported at every slice, for every condition alike. We read the slices one at a time, with no summary over them, since the two ends differ from the rest: slice 0 sees no context, having no block before it, and slice 4 feeds only the readout at its own position, so no state there can pass the op on to the query answer. The anchor term asks for the margin, so it checks that the pull landed and tests nothing.

The edit. The projection at every position, at doses γ = 0.25, 0.5, 0.75, 1 (the share of the e₁ component removed). Ex-2.2.21 (E2) set two criteria: the drop in EEM on difference grows with the dose and reaches at least 50% of the way to the target null at full dose, and no other op drops by more than 0.02 in EEM at any dose. Both drops are net of the drop the control shows under the same edit. Beside the two criteria, the selective reach: how far toward the target null the strongest dose that stays within the gate goes.

Landing (new). The target null is the answer distribution of an ideal predictor that has lost difference and nothing else: it weighs the other ops by how well they fit the examples. For each held-out difference context, we measure how far the answer distribution of the model is from the target null, on the clean model and under the full edit, by their total variation distance: the share of probability that would have to move to turn one distribution into the other, from 0 when they match to 1 when they share no answers. The landing is the share of the clean distance that the edit closes, taken as a ratio of means over contexts: one less the mean edited distance over the mean clean distance. A mean of per-context shares would be dominated by the contexts the clean model already answers like the target null.

Beside it we report the KL divergence of the target null from the model, KL(target null ‖ model).¹1 KL(target null ‖ model) is read "the KL divergence of the target null from the model", i.e. averaged over the answers the target null gives, how much less likely the model finds them, on a log scale. KL divergence measures how much one probability distribution differs from another; it is zero when they match, and it is not symmetric, so the order matters. Only this order is finite: the target null puts no weight on colors that no remaining op gives, and the model puts some weight on every color. A context where the model gives a likely answer almost no weight has a very large divergence. It has no upper bound, so a few contexts can dominate its mean, which is why the landing uses the total variation.

Alignment against the evidence (new). Whether the anchor grades with the evidence: the measurement of (b) in the D2.2 design and of the grading item on the backlog. (The edit criterion above uses "grades" in another sense: the drop grows with the dose.) At each answer in a difference context, α against the posterior on difference given the pairs up to and including that answer (an answer is one color token). That covers the example answers, where ex-2.2.21 found the anchor, and the query answer, where the state already holds the answer, so its posterior counts the query pair too. Every context gives one point per answer, so even the fixed-count corpus has several levels of evidence; k-mixed also contributes the levels of the other counts.

Localized by depth (E1)¶

We look at the slice conditions beside the baseline (hinge) condition, on four measures: the task score net of the control, the op margin at each slice, whether the edit meets the two criteria (the drop on difference grades with the dose, and the other ops stay within the gate), and the selective reach. The question is whether a restriction raises the task score, as leaving out the embedding did on the uncapped pull in ex-2.2.21, while editing as selectively as the hinge condition, and whether the matched weight changes the answer. Ex-2.2.21's uncapped no-emb condition is shown beside its capped twin.

The figure below shows the task score for every condition, so E2 and H1 refer back to it.

What we saw. The task score varies more from seed to seed than from condition to condition. At model seed 700 every anchored condition with three examples scores a little above the control, and at 701 most score near it. At 702 the hinge, no-emb, cap-0.9, cap-0.95, and whole-line conditions end well below the control, and the others end near it (on whole-line, so does the run at 701). So the seed means mostly say whether the run at model seed 702 took the slow path it took in ex-2.2.21 (E5 follows it through training). Of the slice sets, only no-emb at the plain weight put that run on the slow path; on every other slice set it trained as quickly as the runs at the other seeds.

The edit is next: the drop on difference and on the worst other op as the dose grows.

On the hinge condition the edit grades and stays within the gate. Every restriction lets the edit spill onto other ops past the gate, at both weights. The spill is largest on the slice sets that leave out the last slice at the plain weight and on middle-matched, and smallest on no-emb. The matched weight lowers it when the last slice is left out alone and raises it on the other two slice sets. Three conditions miss the grading criterion (no-emb, no-emb-matched, and middle-matched), each because the drop dips between the two strongest doses by a little more than the tolerance of 0.01, while still reaching half the way to the target null.

Taken run by run, hinge is less tidy than its seed mean, since one of its three runs spills past the gate (E2 has more). But the restrictions spill more often: only one of their 18 runs stays within the gate.

The op margin (below) is about as large as on hinge at the middle slices. Where the last slice is left unpulled, the margin there is lower than on hinge. At the embedding it is higher on most restricted conditions, whether that slice is pulled or not.

The alignment at the example answers shows the states of the other ops as well as of difference.

On difference contexts the alignment rises through the stack on every slice set, and sooner on the restricted conditions than on hinge. So do the states of other ops: at the first two slices they sit further along e₁ than on hinge, most of all on the slice sets that leave out the embedding. no-last-matched is the exception, close to hinge throughout.

Various pull caps (E2)¶

The same four measures over the four caps: 0.8 (the hinge condition), 0.9, 0.95, and no cap (ex-2.2.21's whole-line condition). If the selectivity falls off gradually with the cap, a cap near where it crosses the gate gives the most anchor that still edits cleanly. If it drops at one cap, the step is where to stop. At three seeds a gradual fall smaller than the seed range would look flat, and the uncapped condition only just missed the gate in ex-2.2.21, so a flat result would say the cap matters less than the seeds vary.

What we saw. The selectivity does not fall off steadily with the cap. On the seed means, the caps of 0.8 and 0.9 and the uncapped condition stay within the gate, and the cap of 0.95 is past it. That comes from one run: at model seed 700 the edit on cap-0.95 takes another op down by many times the gate, and reaches almost none of the way to the target null within it. The hinge run at the same seed also spills past the gate on its own, though the seed mean stays within it. The uncapped condition, which spilled just past the gate over five seeds in ex-2.2.21, stays within it at these three. Taken run by run, the order changes: two of the three cap-0.9 runs go a little past the gate, and every uncapped run stays within it. So the cap does not put the spills in order: runs past the gate turn up at 0.8, 0.9, and 0.95, and the uncapped condition, within the gate at all three of these seeds, went past it on its seed mean over five seeds in ex-2.2.21.

Past the embedding, the op margin is a little higher at the caps of 0.9 and 0.95 than at 0.8, and the uncapped condition sits just below 0.95. The task score follows hinge at every cap, with the run at model seed 702 on the slow path (E1), and on the uncapped condition the run at 701 as well.

The replicate (post hoc). The three replicate runs of hinge were trained for H2, and the edit criteria can be scored on them too. On their seed means the edit misses both criteria. It spills past the gate, and again the spill comes from one run: at model seed 703 the edit takes darken down by many times the gate. It also misses grading, on every replicate run, in the same way as the slice conditions that miss it: the drop on difference reaches nearly its full size by half the dose, then dips at the stronger doses by a little more than the tolerance. The table below sets the six hinge runs side by side. The edit stays within the gate on four of them.

Mixed counts keep the recipe near its ceiling (H1)¶ ~ Partial

What we expect. On ex-2.2.21's held-out set, which has three examples per context, we expect the k-mixed condition to fall short of the hinge condition by less than 0.015 in EEM, paired by seed: a pass. A shortfall above 0.015 but inside the seed band would be a partial pass, since three seeds could not tell it from noise, and a larger one a miss. A gain would also be a pass. If the seeds disagree about the direction by more than the tolerance each way, the result would be outside the plan, and the verdict would be Unresolved.

At each example count we also report the skill of k-mixed, the share of the way from the floor (a predictor that ignores the examples) to the ceiling, with no gate. A count where the skill falls well below the others would say the model has not learned that length, which matters for the alignment against the evidence (E3).

What we saw. k-mixed falls short of hinge at every seed, by 0.08 to 0.12 in EEM, and it is below the control at every seed too (E1). The mean shortfall is above the tolerance and just inside the seed band. The band is wide mostly because both conditions score lower at model seed 702, on the slow path described in E1.

The skill of k-mixed rises steadily with the number of examples, so no count stands out as one the model failed to learn. hinge and the control, trained on three examples alone, score higher than k-mixed at three examples and lower with one or two. With four or five, the control stays higher and hinge falls below. That fall comes from one run: at model seed 701 the skill of hinge drops to 0.35 at four examples and to -0.03 at five (the floor is 0), while its other two runs lose much less. That run is not the slow one, so it seems that how well a model trained on three examples copes with longer contexts varies from seed to seed.

The anchor and the posterior (E3)¶

Whether α at the answers rises with the posterior on difference. The label is the same for every difference context however well its examples fit, so if α grades anyway, the anchor follows what the model has inferred and not just the label. Evidence builds up along a context, so the posterior also rises with the position of an answer; to tell the two apart, we also compare contexts at the same answer index, where the posterior still differs from one context to the next. We look at it on k-mixed, and on the hinge and whole-line conditions of ex-2.2.21 at three examples. This is a first look that would shape a later prediction, so it has no gate.

First, how the answers spread over the posterior. Each point is one answer of a held-out difference context.

What we saw. About half the answers sit above 0.9, and the rest fall on a few steps lower down. The two corpora give nearly the same spread, so for this measurement mixing the counts added little. The query answers sit almost all in the top bin, since the query pair is counted too, so the curves below are most informative at the example answers.

Past the embedding, α at the example answers rises with the posterior on every anchored condition, and the control stays flat near zero. The rise is uneven: the bin from 0.6 to 0.8 sits above the top bin. Most of the points in that bin are answers of the first example, and the next figure suggests why that matters. k-mixed sits lower than the two three-example conditions in most bins past the embedding, and at the embedding itself.

The posterior also rises along a context, so the next figure plots each answer index separately.

At a fixed index, α rises with the posterior at every example answer. At the same posterior, it sits higher at earlier answers: at the first answer, α in the bin from 0.6 to 0.8 is about as high as it gets anywhere. So the bump in the pooled figure above seems to come from mixing answer indices, with the first answers crowded into that one bin. The first answer has few posterior levels: about two thirds of its points sit at 0.69 and most of the rest at 0.05, so its two middle points rest on a few dozen contexts each, and its line rises in steps.

The edit lands on the target null (H2)¶ ✗ Miss

What we expect. In ex-2.2.21, after the fact, the edited hinge condition answered difference contexts about as the target null does. If that holds, we would not need to train a designed fallback, since the anchor alone gives the edit a predictable destination. The prediction came from looking at ex-2.2.21's hinge runs, so scoring those same runs would show little. We score it on the replicate, three hinge runs at fresh seeds, and report the stored hinge runs and every other condition beside it.

At full dose, with the edit at every position, on held-out difference contexts, we expect the edit to close at least 50% of the distance from the target null that the clean model has, in total variation, as a ratio of means over contexts. That is the same share as the edit criterion of ex-2.2.21, which asks for half the way to the target null in EEM, and on the same scale: both run from the clean model (0) to the target null (1). The landing asks for more, since mass that leaves the difference answer counts toward it only where it arrives on answers the target null gives. Closing that share in every seed of the replicate would be a pass, and falling short in every seed a miss. If some seeds pass and others fall short, the result would be outside the plan, and the verdict would be Unresolved.

What we saw. The edit moves every replicate run a little under half of the way to the target null, and none reaches the gate. The stored hinge runs land in the same place, and the runs of the other anchored conditions land near them, apart from k-mixed at model seed 702; the control hardly moves. The KL divergence falls further, to about a quarter of its clean value. KL grows large wherever the model puts next to nothing on an answer the target null gives. The clean model, sure of difference, does that on many answers, which is why its KL is large. So it seems that under the edit far fewer such answers remain.

The confusion matrices show where the answers go, at three levels of how sure the context is about its op.

In the difference row, the edited model puts about as much mass on the answers of each other op as the target null does, in every bin. The mass left on the answers of difference itself, on the diagonal, falls close to the little the target null puts there. So, pooled over contexts, the edit puts the mass about where the target null does.

The matrices and the landing measure different things. A matrix adds up the mass on the answers of each op over all the contexts in a bin, so a model that puts too much on one op in some contexts and too little in others can still match the target null in total. The landing measures the distance on each context and then averages, so those mismatches add up instead of cancelling. E4 looks at where they fall.

In the rows of the other ops, the edit moves a little mass off the true op and onto the others in the two surer bins, where the target null puts almost none. That loss is in the EEM of the other ops, and E2 scores it net of the control. In the two surer bins, the largest entries off the diagonal are between sat-hsv and value-hsv, which the clean model already confuses with each other at about the same level; the edit adds a little to them.

Where the distance from the target null remains (E4, post hoc)¶

H2 scores the distance from the target null context by context, and the confusion matrices, which pool over contexts, make the edited model look closer to the target null than the landing says. So here we split the distance by the answers it falls on: answers another op could give, answers only difference gives, and colors no op gives.

What we saw. On the clean model, about 41% of the distance is mass on answers only difference gives. The edit removes nearly all of it and moves little onto colors no op gives. What remains, about 88% of the edited distance, is on answers another op could give: the edited model puts its mass on the right kind of answer and shares it among those answers differently from the target null.

The sharing partially follows the examples. The mass the edited model puts on the answers of each other op rises and falls with the mass the target null puts there, though loosely (the correlations over contexts are in the table below).

For scale, the clean model is some way from an ideal predictor too. On the same contexts it is about 0.17 from the ideal predictor behind the Bayes ceiling (which weighs all seven ops by how well they fit the examples), in the same measure, which is about 39% of the distance that remains under the edit.

Training trajectories (E5, post hoc)¶

The run at model seed 702 ends low on several conditions (E1), as it did in ex-2.2.21. To see when it parts from the others, the figure below follows the task score through training for every run, on a subsample of the held-out set, with the run at that seed highlighted.

What we saw. Most runs rise quickly to a first plateau, stay on it for some tens of epochs, and then rise again, at a time that varies from run to run. On the conditions where the run at model seed 702 ends low, that run never makes the second rise and climbs slowly from the plateau for the rest of training. On whole-line the run at model seed 701 does the same, and on the control one run makes the second rise late. On the restricted conditions, other than no-emb, every run makes the second rise and the three end together, as do the runs of the uncapped no-emb. All three replicate runs of hinge make it, one ending a little lower than the other two.

The op margin at the last slice, below, shows whether the slow runs also hold the anchor less firmly.

The margin shows no matching split: on most conditions the slow run ends with the others or above them. k-mixed is the exception, where the run at model seed 702 also ends with the lowest margin.

Decision¶

Three choices for the recipe: which slices the pull acts on, and at what weight; the cap; and whether the corpus mixes example counts. The tables below score every candidate (the conditions of this scout and the ex-2.2.21 conditions they pair with) on the criteria frozen with the plan, and the choice follows them.

What we chose. For the recipe, one of the three choices changes:

• Slices: every slice stays in the pull, at the plain weight. Every restriction let the edit spill past the gate more often than hinge did (E1), at either weight. Which slices the pull needs is still open; pooling over slices, so that training can choose where to hold the op, is the next idea to try.
• Cap: none. Runs past the gate turn up at every cap from 0.8 to 0.95, and hinge stays within the gate on four of its six runs (E2). So at three seeds the cap seems not to buy the selectivity it was meant to, and leaving it out removes a setting.
• Counts: three examples per context. k-mixed falls a little short of hinge (H1), its α sits lower (E3), and three examples already give α enough spread to grade at a fixed answer index (E3).

Before confirming these at fresh seeds, we would like to understand the seeds better. One seed takes a slow path on most anchored conditions (E5), and along the caps the spills past the gate come from single runs, all of them runs that trained quickly.

Discussion¶

It seems the recipe of ex-2.2.21 was already close to the best these three changes allow, and what most limits the next step is how much the runs vary from seed to seed.

Leaving slices out of the pull was meant to put the anchor where an inferred op lives. Instead, every restriction let the edit spill onto other ops. At the first slices the states of other ops sat further along e₁ than on hinge, most of all where the embedding was left out (E1). Leaving a slice out of the pull also leaves out the term that pushes other ops off e₁ there, so those states may simply have been free to drift onto the axis. Keeping that term on every slice while restricting the pull would tell the two apart.

The cap did less than we hoped. Ex-2.2.21 suggested that a cap keeps the edit selective, but with the runs taken one at a time, spills turn up at every cap, and at these seeds not on the uncapped condition. Along the caps, the spills come from single runs, and only from runs that trained quickly. A run on the slow path may not yet have reached the stage of training where spilling happens, which is part of why the seeds come first.

The edited model looks better pooled over contexts than one context at a time. Counted by op, the mass leaves difference and lands on the other ops about where the target null puts it (H2). Context by context, the edit gets a little under half of the way. Nearly all of what remains is on answers another op could give, shared among them differently from the target null, and part of it is the gap the clean model already has from an ideal predictor (E4). So "behaves like the target null, pooled over contexts" seems a fair description of the edited model. We don't yet know how close a model trained without ever seeing a difference context would come to the target null, which would say how much of the remaining distance any model would leave.

The anchor follows the evidence. At the example answers α rises with the posterior on every anchored condition, and at a fixed answer index too (E3). At the same posterior, α is higher at earlier answers, which we did not expect and can't yet explain.

Method¶

The corpus. The fixed-count conditions train on ex-2.2.21's corpus of three-example contexts. k-mixed trains on a corpus of the same number of contexts, built by the same generator, with the number of examples drawn uniformly per context from 1, 2, 3, 4, 5. Two whole contexts at five examples take 76 tokens, inside the window (block size) of 96. The held-out set for k-mixed has 2,000 contexts per op at each count. Every condition is also scored on ex-2.2.21's held-out set, so k-mixed compares with its reference on the same contexts.

The edit and the landing. The suppression pass is ex-2.2.21's, at every position only. The landing measurements compare the answer distribution of the edited model over the color vocabulary at the query = with the target null from the posterior over the other ops (sca.data.incontext.target_null).

Budget. Planned at under $5: 30 runs at about $0.13 each on an L4 (the cost of ex-2.2.21's runs), with the scoring passes. The experiment cost $6.69 in all.