Setting the terms by slice¶

We held the anti-subspace weight high on the embedding and block 1 only, and at the recipe level on the pulled blocks. This kept most of the selectivity of a high hold everywhere, and it gave back the removalRemoval How much of difference the edit takes out: the net drop in its score under the edit, as a share of the way from the clean score to the target null. that the high hold cost.

We also pooled the pull over slices as well as positions. On most seeds the pull then settled on the last block. But on one seed it settled on a position the prediction does not use, and there the op could not be removed. On one of the block-4 seeds, a softer pool over slices (a higher $\tau_\ell$) let the earlier blocks align too, and removal and spillSpill The largest net drop in score on another op under the edit. The selectivity criterion is a spill of at most 0.02. stayed about where they were.

The late-pull experiment pulled difference toward e₁ at blocks 2 to 4 only, with the anti-subspace termAnti-subspace term The companion to the pull: the mean of α² over every position and slice it acts on, which asks the states as a whole to stay off e₁. Its weight $\lambda_{\bar s}$ starts at 0.25 and anneals to a level it then holds, the hold. on every slice (the deep arm). Holding the anti weight at 0.2 for all of training, in place of the recipe hold of 0.03, kept the color embedding table and block 1 off e₁ on the other ops and brought the spill within the criterion on every seed. But it removed about a tenth less of the op.

A second review of those runs found that the hold also counters the pull at the pulled blocks, which may account for the lost removal. So we set the two terms by slice, in two arms of 4 runs each at 400 epochs:

split-anti: the deep pull, with the anti weight held at 0.2 on the embedding and block 1 and at the recipe hold of 0.03 on blocks 2 to 4.
pool-slices: the same anti weights, with the pull on blocks 1 to 4 pooled over slices as well as positions, so that it may choose where along the stream to align.

A second round adds 4 runs of pool-slices at seed 701, with the pool over slices at a temperature of its own, $\tau_\ell$, to ask whether a softer pool would settle less of the pull on block 4.

Observations¶

Removal and spill (E1): with the anti weight split by slice, removalRemoval How much of difference the edit takes out: the net drop in its score under the edit, as a share of the way from the clean score to the target null. is back to that of deep at the recipe hold, and the spillSpill The largest net drop in score on another op under the edit. The selectivity criterion is a spill of at most 0.02. stays within the criterion at every dose on 3 of 4 seeds. The fourth spills a little more than the criterion allows, at the full dose only.
Where the anti weight counters the pull (E2): the split keeps the alignmentAlignment How far a state points along e₁: the cosine between the state and e₁, which is 1 on the anchor direction and 0 off it. at the fitting exampleFitting example An example whose answer is one that difference could give on its operands, whatever op the context is under. answers where the recipe hold has it, and keeps block 1 and the color embedding table nearly as far off e₁ on other ops as the high hold does.
Where the pooled pull settles (E3): pooled over slices, the pull settles on block 4 on three seeds, and there the edit removes the op about as selectively as the split does. On seed 700 the pull settled on the query answer alone, a position the prediction does not read, and the op is not removable there.
A softer pool over depth (E4): on seed 701, raising $\tau_\ell$ leaves block 4 where it was and lets blocks 1 and 2 align about as far as in split-anti. Removal and spill move around from one temperature to the next with no trend, and every run stays within the criterion.

Scope¶

This is an exploratory study, with no preregistration and no gate. Four seeds per arm resolve the large differences between the late-pull holds, but not whether a single run over the criterion is typical. E4 has one run per temperature, so it can show a change in where the pull settles, but not a difference in removalRemoval How much of difference the edit takes out: the net drop in its score under the edit, as a share of the way from the clean score to the target null. of the size that seeds differ by.

Each run shares its seed, and so its initialization, batches and label draws, with a late-pull deep run at each hold and an ex-2.2.23 control. So a difference between arms at one seed comes from the change of terms rather than the initialization or the batches.

The peak anchor weightAnchor weight The weight of the pull in the loss. It ramps up to its peak, holds, and anneals away at the end of training. $\lambda_a$ (0.1), the pooling temperaturePooling temperature How sharply the pull concentrates where alignment is cheapest. Each labeled context is pooled over its positions at $\tau_p$ (low: nearly all of the pull goes to the best-aligned position; high: it spreads evenly), and in pool-slices and E4 also over slices at $\tau_\ell$. $\tau_p$ (0.1), the label, the corpus and the schedules are the recipe values. Both arms keep the recipe anti schedule: 0.25 at the start on every slice, annealing to its hold by epoch 360, then sharing the anchor anneal at the end. The anti weight $\lambda_{\bar s}$ is now written as an absolute number rather than a ratio to the anchor weight, which gives the same schedule here.

The measurements¶

The measurements are those of the late-pull experiment, with the dose scoring of ex-2.2.22 added for every run and its twins:

RemovalRemoval How much of difference the edit takes out: the net drop in its score under the edit, as a share of the way from the clean score to the target null.: the net drop in the score on difference under the edit, as a share of the way from the clean score to the target nullTarget null The score an ideal predictor would get on difference contexts if it had lost that op and answered from the remaining ones.. The edit projects e₁ out of the state at every position, on a set of slices, and at a dose of a quarter, a half, three quarters, or all of the e₁ component.
SpillSpill The largest net drop in score on another op under the edit. The selectivity criterion is a spill of at most 0.02.: the largest net drop on another op. Both are net of the control at the same seed, and the selectivity criterion is a spill of at most 0.02 at every dose.
The alignmentAlignment How far a state points along e₁: the cosine between the state and e₁, which is 1 on the anchor direction and 0 off it. α with e₁ at the example answers, the positions where the anchor sits in this grammar, along training at every slice.
The task score of each op, as expected exact matchExpected exact match The task score, as the chance that a drawn answer agrees with a true one. With drawn answers, the correct answer to a line is spread over two or more colors, so we take the chance that an answer drawn from the model agrees with one drawn from the true distribution. Its ceiling is that same chance for the true distribution against itself (Σq²), below 1 on the ops that round. on held-out contexts, against the control at the same seed.

Removal and spill (E1)¶

Suppose the high $\lambda_{\bar s}$ hold costs removalRemoval How much of difference the edit takes out: the net drop in its score under the edit, as a share of the way from the clean score to the target null. because it counters the pull at blocks 2 to 4. Then holding it high only on the embedding and block 1 should give the removal back. And if the stand-insStand-in Our working hypothesis for what carries the spill, not settled: something token-level that satisfies the pull where no contextual concept can, such as lightness on e₁ in the color embedding table (the lean), or a syntax token embedding on e₁ (a latch). live at those first two slices, the spillSpill The largest net drop in score on another op under the edit. The selectivity criterion is a spill of at most 0.02. should also stay low.

The main measure is removal at the full dose, with the spill at every dose beside it.

Removal grows with the dose in every arm; deep at the recipe hold and split-anti reach about 0.85 to 0.9 at the full dose, and deep at a hold of 0.2 reaches 0.64 to 0.86. Spill stays near zero at every dose for the two arms with a high hold on the first slices, with one split-anti run reaching 0.025 at the full dose, while deep at the recipe hold rises past the criterion from the three-quarter dose on. — **Removal and spill by dose**, the edit at every position and slice. One hairline per run and the seed mean bold. The dashed rule is the selectivity criterion. `deep` and `deep, hold 0.2` are the late-pull runs at the same seeds.

Both hold. On every split-anti run, removal grows with the dose to about what deep removes at the recipe hold (the table below has the ranges).

The spill stays within the criterion at every dose on 3 of 4 seeds. On seed 701 it goes over the criterion at the full dose only (0.025, on mix), the same seed and op where deep at the recipe hold spilled least. The high hold everywhere is the only arm within the criterion on every seed.

arm	removal	within criterion	spill, any dose	raw spill	landing	task change
`deep`	0.88 (0.86 to 0.90)	0 of 4	0.090 (0.038 to 0.171)	0.122 (0.048 to 0.207)	0.47 (0.42 to 0.50)	−0.003 (−0.006 to 0.001)
`deep, hold 0.2`	0.77 (0.64 to 0.86)	4 of 4	−0.001 (−0.004 to 0.001)	0.006 (0.003 to 0.014)	0.48 (0.41 to 0.55)	−0.005 (−0.018 to 0.003)
`split-anti`	0.87 (0.84 to 0.88)	3 of 4	0.006 (−0.005 to 0.025)	0.015 (0.001 to 0.036)	0.49 (0.44 to 0.51)	−0.006 (−0.022 to 0.003)
`pool-slices`	0.63 (−0.04 to 0.90)	3 of 4	0.006 (−0.007 to 0.034)	0.029 (0.011 to 0.061)	0.36 (0.04 to 0.51)	−0.004 (−0.011 to 0.003)

Removal, spill, landing and task, seed mean and range over 4 runs. Removal is at the full dose; "spill, any dose" is the largest net spill over the four doses, and "within criterion" counts the runs where it is at most 0.02. The raw spill is the largest drop on another op at the full dose without subtracting the control. The landing is the share of the distance to the target null that the full edit covers, as total variation on difference contexts. The task change is the mean score over ops, less that of the control. pool-slices is the subject of E3.

The edit lands about as close to the target nullTarget null The score an ideal predictor would get on difference contexts if it had lost that op and answered from the remaining ones. in split-anti as in both deep arms. So the extra removal seems to be the op itself going away, rather than answer mass scattered onto other ops.

The task change is small, as in deep. One split-anti run (seed 702) has the lowest task score against its control of any run so far. The loss is spread across all ops rather than concentrated on difference, which suggests seed variation more than a cost of the terms. Four seeds cannot separate the two.

The slices the edit has to cover are the same as in late pull. As in deep, editing blocks 2 to 4 alone removes nearly everything on two seeds and less than half on the other two, and editing block 1 alone removes most of the op on every seed.

So on four seeds the split gets nearly all the selectivity of the high hold with none of its cost in removal.

Where the anti weight counters the pull (E2)¶

Did the removalRemoval How much of difference the edit takes out: the net drop in its score under the edit, as a share of the way from the clean score to the target null. come back for the reason we expected, the alignmentAlignment How far a state points along e₁: the cosine between the state and e₁, which is 1 on the anchor direction and 0 off it. at the pulled blocks restored with block 1 and the table still off e₁ on the other ops?

At the example answers of difference contexts, blocks 2 to 4 reach about 0.5 early in every arm, then climb to about 0.65 in deep and split-anti and stay near 0.55 at the high hold. At block 1 on the other ops, deep drifts up through the second half to about 0.3, while the high hold and split-anti stay near 0.1. — **Alignment at the example answers along training.** Left: α at the example answers of `difference` contexts, averaged over blocks 2 to 4. Right: α at block 1 at the example answers of the other ops. One hairline per run and the seed mean bold. The strip below is the anti weight: `deep` at the recipe hold (solid) and at 0.2 (dashed). `split-anti` follows the dashed line on the embedding and block 1 and the solid line on blocks 2 to 4.

It did. At blocks 2 to 4, split-anti follows deep at the recipe hold: the alignment reaches about a half early in training and climbs through the second half, where the high hold everywhere holds it flat. At block 1 on the other ops it follows the high hold, and stays low where deep drifts up through the second half.

arm	fitting, `difference`	non-fitting, `difference`	fitting, other ops	block 1, other ops	slope in the table
`deep`	0.96 (0.95 to 0.96)	0.11 (0.09 to 0.14)	0.92 (0.90 to 0.94)	0.32 (0.23 to 0.41)	−0.26 (−0.30 to −0.18)
`deep, hold 0.2`	0.84 (0.76 to 0.90)	0.02 (0.01 to 0.04)	0.74 (0.66 to 0.83)	0.09 (0.04 to 0.15)	−0.04 (−0.12 to 0.01)
`split-anti`	0.94 (0.93 to 0.95)	0.09 (0.06 to 0.12)	0.90 (0.89 to 0.93)	0.13 (0.10 to 0.17)	−0.08 (−0.11 to −0.03)

Where the anchor sits at the end of training, seed mean and range over 4 runs. The first three columns are α at the example answers averaged over blocks 2 to 4, on held-out contexts, split by whether the example fits difference (whether that op could give the answer it shows) and by the op of the context. "Block 1, other ops" is α at block 1 at the example answers of the other ops, on the probe contexts. The slope is the lean of the color embedding table: the e₁ component of the color embeddings against lightness. The controls are at about 0.04 in the first three columns and lean either way, up to about ±0.25.

Split by whether an example fits difference, the alignment at the fitting answers is back to nearly the recipe-hold level, so the high hold no longer counters the pull at the pulled blocks.

The non-fitting answers are back near the recipe-hold level too. These answers align whatever op the context is under, so this alignment does not mark the op. The high hold everywhere had cleared it, and the split leaves it in place, but it does not show up in the spillSpill The largest net drop in score on another op under the edit. The selectivity criterion is a spill of at most 0.02..

The color embedding table leans about a third as much as in deep, a little more than at the high hold, and on every seed it leans the way the anchored runs lean. Block 1 on the other ops is a little above the high hold too. Both fit the small spill that remains on one seed.

Where the pooled pull settles (E3)¶

Pooled over slices as well as positions, each slice gets a share of the pull that grows steeply with how well aligned it already is. So the pull should settle on whichever slice aligns most readily. The embedding is not in the pool: the pull acts on blocks 1 to 4, and only the anti-subspace termAnti-subspace term The companion to the pull: the mean of α² over every position and slice it acts on, which asks the states as a whole to stay off e₁. Its weight $\lambda_{\bar s}$ starts at 0.25 and anneals to a level it then holds, the hold. acts on the embedding. The figures show it for comparison.

Which slice is that, and does the edit at every slice still remove the op?

On seeds 701 to 703 the pooled pull leaves the example answers at about 0.2 at block 1, rising to 0.65 to 0.75 at block 4, close to split-anti at the same seed; on seed 700 the example answers stay near zero at every slice while the query answer reaches 1 at blocks 3 and 4. — **Where the pooled pull settles, by seed.** α at the end of training on `difference` probe contexts, at each slice: at the example answers (filled markers) and at the query answer (open markers), for `pool-slices`. The hairline is `split-anti` at the example answers, at the same seed.

On three seeds the alignmentAlignment How far a state points along e₁: the cosine between the state and e₁, which is 1 on the anchor direction and 0 off it. is largest at blocks 3 and 4, with block 2 nearly as high and block 1 low. That profile is much like that of split-anti, though block 1 aligns less.

On those seeds the spillSpill The largest net drop in score on another op under the edit. The selectivity criterion is a spill of at most 0.02. is within the criterion at every dose, and the edit removes the op as well as in split-anti on two of them and about a tenth less on the third (seed 702).

seed	α, block 3	α, block 4	removal	block 1 only	blocks 2 to 4 only	spill, any dose
700	0.04	0.02	−0.04	0.04	0.02	0.034
701	0.62	0.64	0.90	0.58	0.94	−0.007
702	0.72	0.75	0.76	0.78	0.01	0.001
703	0.63	0.65	0.89	0.76	0.93	−0.004

The pooled pull, by seed. α at the example answers of difference probe contexts at blocks 3 and 4. Removal at the full dose of the edit at every slice, then of the edit on block 1 alone and on blocks 2 to 4 alone. The spill is the largest net spill over the four doses of the edit at every slice.

On seed 700 the example answers stayed near zero at every slice, and the edit removes nothing. The query answer aligned instead, reaching α of 1 at blocks 3 and 4 (the state at the answer token of the query has already seen the answer, so it can show the op).

No other position of the probe contexts reaches more than 0.11 at any slice. The query answer aligns in every arm, since it is part of every labeled context, but in pool-slices on this seed it took the whole of the pull.

Along training the example answers rose to about a fifth early on, then fell back as the query answer took over. The prediction of the query answer is made at the query =, before that position, so an edit there cannot reach it.

Compared with split-anti, this arm changes two things: the pull is pooled, and block 1 joins the pulled slices. But block 1 took almost none of the pull on this seed, so the pooling is the likelier cause.

A softer pool over depth (E4)¶

With the slices pooled at the recipe temperature, the pull may put nearly all its weight on whichever slice is ahead, and on three seeds that was block 4. A softer pool over slices spreads the pull back over depth. At a high enough temperature, it becomes the mean of one pool per slice.

So we trained pool-slices again at seed 701, where it had settled on block 4, with $\tau_\ell$ at 0.2 to 1.6, each double the last, and the pool over positions still at $\tau_p = 0.1$. When $\tau_\ell = \tau_p$ the two pools are the same as the one joint pool of E3, so the pool-slices run is the first rung of the ladder.

At every temperature the pull reaches the same level at block 4; softening the pool raises blocks 1 and 2 at the example answers to about where split-anti has them, and the query answer aligns at blocks 3 and 4 throughout. — **Where the pull settles as the pool over slices softens**, seed 701. α at the end of training on `difference` probe contexts at each slice: at the example answers (left) and at the query answer (right), one line per temperature $\tau_\ell$ of the pool over slices, from the recipe value (`pool-slices`, lightest) to the softest. The hairlines are the per-slice arms at the same seed: `deep`, `deep, hold 0.2`, and `split-anti` (dotted).

Softening the pool did not take the alignmentAlignment How far a state points along e₁: the cosine between the state and e₁, which is 1 on the anchor direction and 0 off it. away from block 4. It stays at about 0.64 at every temperature, close to the per-slice arms at this seed.

The earlier blocks change instead. At $\tau_\ell = 0.1$, block 1 is at 0.22 and block 2 below the blocks after it. From the first softer step on, both rise, and the profile over slices looks like that of split-anti at this seed. So on this seed the sharp pool had not latched onto block 4 so much as held back the blocks before it.

The query answer shifts the same way, with blocks 1 and 2 higher at the softer temperatures. The example answers keep their alignment at every temperature, so this seed shows nothing of the seed 700 outcome. One seed cannot say whether a softer pool would have kept that seed on the example answers.

τ_ℓ	α, block 1	α, block 4	removal	block 1 only	blocks 2 to 4 only	spill, any dose	task change
0.1	0.22	0.64	0.90	0.58	0.94	−0.007	0.002
0.2	0.35	0.64	0.83	0.65	0.91	−0.006	−0.001
0.4	0.38	0.64	0.90	0.94	0.72	−0.007	0.004
0.8	0.41	0.65	0.74	0.74	0.76	0.002	0.002
1.6	0.42	0.65	0.87	0.82	0.90	0.015	−0.006
`deep`	0.51	0.65	0.88	0.88	0.92	0.038	0.001
`deep, hold 0.2`	0.32	0.60	0.84	0.63	0.88	−0.002	0.002
`split-anti`	0.38	0.65	0.88	0.84	0.92	0.025	0.000

The pooled pull by temperature of the pool over slices, seed 701, with the per-slice arms at the same seed below. α at the example answers of difference probe contexts at block 1 and block 4. Removal at the full dose of the edit at every slice, then on block 1 alone and on blocks 2 to 4 alone. The spill and task change are as in E1. The first line is pool-slices, whose pool over slices is at the recipe value. Resampling the held-out contexts moves removal by about 0.005 (one standard deviation) on every run.

Every run on this ladder is within the criterion at every dose, and the task change is small. The largest spillSpill The largest net drop in score on another op under the edit. The selectivity criterion is a spill of at most 0.02. rises a little at the two softest temperatures, toward that of split-anti at this seed.

RemovalRemoval How much of difference the edit takes out: the net drop in its score under the edit, as a share of the way from the clean score to the target null. under the full edit is 0.74 to 0.90 across the temperatures, with no trend, and the edit on block 1 alone or on blocks 2 to 4 alone moves around even more. That spread is far larger than the measurement error, so the runs end up as different models even though they share a seed.

The spread is also wider than that of the three per-slice arms at this seed (0.84 to 0.88), and about as wide as that of pool-slices over its three block-4 seeds in E3. So the pooled pull may make the outcome more sensitive to small changes in training.

Discussion¶

On the embedding and block 1, the high $\lambda_{\bar s}$ hold keeps the stand-insStand-in Our working hypothesis for what carries the spill, not settled: something token-level that satisfies the pull where no contextual concept can, such as lightness on e₁ in the color embedding table (the lean), or a syntax token embedding on e₁ (a latch). off e₁ (if stand-ins are what carry the spillSpill The largest net drop in score on another op under the edit. The selectivity criterion is a spill of at most 0.02.). On the pulled blocks, it counters the alignmentAlignment How far a state points along e₁: the cosine between the state and e₁, which is 1 on the anchor direction and 0 off it. at the fitting exampleFitting example An example whose answer is one that difference could give on its operands, whatever op the context is under. answers, which fits where the removalRemoval How much of difference the edit takes out: the net drop in its score under the edit, as a share of the way from the clean score to the target null. went. Splitting the weight keeps the first effect and removes the second, as the second review expected.

The one split-anti seed over the criterion spills on the same op where deep spilled least, and a little more of block 1 and the table sit on e₁ than at the high hold everywhere. So the split may trade a small amount of selectivity for the removal it gives back. If that trade is typical, a hold a little above 0.2 on the first two slices, or the hold of 0.2 with a higher one on block 1, would be the place to look.

Where the example answers aligned early, the pooled pull settled on block 4 and the result was much like the split. Where the query answer got ahead, the pull settled there alone, after the prediction it would matter for. To work as a recipe, the pooled pull at $\tau_\ell = 0.1$ would need the query answer left out of its pool, which we could not do in production. Even then, the anchor would depend on which position aligns first.

On the one seed tried, a softer pool over slices let blocks 1 to 4 all align, with block 4 still the furthest along, close to the per-slice pull of split-anti. So the softer pool takes away the concentration over slices, which was the point of pooling over them.

Glossary¶

Alignment
α: How far a state points along e₁: the cosine between the state and e₁, which is 1 on the anchor direction and 0 off it.
Anchor weight
$\lambda_a$: The weight of the pull in the loss. It ramps up to its peak, holds, and anneals away at the end of training.
Anti-subspace term: The companion to the pull: the mean of α² over every position and slice it acts on, which asks the states as a whole to stay off e₁. Its weight $\lambda_{\bar s}$ starts at 0.25 and anneals to a level it then holds, the hold.
Fitting example: An example whose answer is one that difference could give on its operands, whatever op the context is under.
Pooling temperature
$\tau_p$, $\tau_\ell$: How sharply the pull concentrates where alignment is cheapest. Each labeled context is pooled over its positions at $\tau_p$ (low: nearly all of the pull goes to the best-aligned position; high: it spreads evenly), and in pool-slices and E4 also over slices at $\tau_\ell$.
Removal: How much of difference the edit takes out: the net drop in its score under the edit, as a share of the way from the clean score to the target null.
Spill: The largest net drop in score on another op under the edit. The selectivity criterion is a spill of at most 0.02.
Stand-in: Our working hypothesis for what carries the spill, not settled: something token-level that satisfies the pull where no contextual concept can, such as lightness on e₁ in the color embedding table (the lean), or a syntax token embedding on e₁ (a latch).
Target null: The score an ideal predictor would get on difference contexts if it had lost that op and answered from the remaining ones.