What the edit does on the recipe of record¶

The D2.2 design doc lists three claims that the recipe of record has no measurement for yet. None needs new training.

• (d) The side effects are bounded, and the boundClean bound From the clean pass of the anchored model, before any edit: arcsin of the 99th percentile of |α| over the states of the six other ops at a slice and position. At the answers it counts only those that do not fit difference. It follows H4 of ex-2.2.1. can be stated in advance from the clean model. That the edit turns a state by arcsin of its alignmentAlignment How far a state points along e₁: the cosine between the state and e₁, which for a unit-length state is its component along e₁; 1 on the anchor direction and 0 off it. holds by construction. What has to be shown is downstream: the blocks after an edit put no more e₁ back on the other ops than the clean model has there. This is the write bound.
• (b) Removing the op is graded: the drop on each context follows how much its answer depended on difference, as the target null predicts context by context.
• (c) The edited model answers in a predictable way: once difference is removed, it answers as the target null does, weighing the other ops by how well they fit the examples.

This study measures all three on the eight falloff runs of split-anti-seeds, the experiment that set the current recipe of record. It uses the dose scoring that experiment stored, plus one new pass over the held-out contexts. That pass records the alignment with e₁ at every slice and position, clean and under the edit.

Observations¶

• How far the edit turns the other ops (E1): the writeWrite How far the edit turns a state: arcsin |α| of the alignment the state arrives with, taken at the 99th percentile over a group of states. At the embedding that is the clean α. At a later slice it is the α left after the block before has read the states the edit turned at the slices below. It is taken before the edit at that slice, which leaves no e₁ behind. stays within the clean boundClean bound From the clean pass of the anchored model, before any edit: arcsin of the 99th percentile of |α| over the states of the six other ops at a slice and position. At the answers it counts only those that do not fit difference. It follows H4 of ex-2.2.1. at every slice past the embedding, at most 1.08 times it (seed mean). From slice 2 on, the other ops arrive with less e₁ than the clean model gives them, and less than the control has. The bound is widest at the answers, and at the example answers it is widest at slice 1. Example answers of other ops that fit difference are evidence for it; they sit on e₁, and the edit turns them most.
• Whether the drop follows what each context needed (E2): the edit takes about the same share of the way to the target nullTarget null Where a removal should land: what an ideal predictor that had lost one op and nothing else would answer. It is the Bayes posterior over ops given the examples, with the removed op set to zero and the rest renormalized, so it still weighs the other ops by how well they fit. on contexts that barely needed difference as on those that needed it most. Some contexts of other ops have examples that point at difference; there the edit gives back some of what those examples cost, as the target null says it should.
• Where the edited answers fall short (E3): almost all the mass leaves the answers only difference gives. It lands on the answers of the other ops, in about the right overall proportions. What is left is mostly how the edited model weighs those ops context by context, and those weights are no flatter or sharper than the target null on the whole. The distance leftDistance left The total variation distance between the edited answers and the target null, on difference contexts. is about 2.4× the distance between the clean model and the Bayes predictorBayes predictor The best answer given the examples: it weighs the answers of each op by the posterior over ops, the probability of each op given the examples. it was trained to match.

Scope¶

This is an exploratory re-analysis, with no preregistration and no gate. It covers the falloff arm of split-anti-seeds (the recipe of record) at model seeds 704 to 711. It also covers the control at each of those seeds from ex-2.2.23, which was trained without the anchor at the same initialization, batches, and label draws.

Every measurement is on the 14,000 held-out contexts of ex-2.2.21, 2,000 per op, each with three examples and a query. The edit is the full edit of the recipe: the whole e₁ component taken out of the state at every position and every slice. E2 and E3 use the full dose only, since split-anti-seeds already showed that the removal grows with the dose.

With eight runs, a pattern seen on every seed is one we can lean on; a difference between seeds is noted and not explained. E3 confirms on the recipe of record what ex-2.2.22 found informally on an earlier recipe, and takes it one step further.

Method¶

The states are unit vectors, and the edit takes the e₁ component out of each one and renormalizes. That turnsWrite How far the edit turns a state: arcsin |α| of the alignment the state arrives with, taken at the 99th percentile over a group of states. At the embedding that is the clean α. At a later slice it is the α left after the block before has read the states the edit turned at the slices below. It is taken before the edit at that slice, which leaves no e₁ behind. each state by an angle set by its alignmentAlignment How far a state points along e₁: the cosine between the state and e₁, which for a unit-length state is its component along e₁; 1 on the anchor direction and 0 off it., arcsin |α|, so how far the edit moves a state can be read off the clean geometry before any edit is made. The first figure draws this, with the boundClean bound From the clean pass of the anchored model, before any edit: arcsin of the 99th percentile of |α| over the states of the six other ops at a slice and position. At the answers it counts only those that do not fit difference. It follows H4 of ex-2.2.1. and the write of E1. That part holds by construction; the second figure shows the part that does not, where the write is taken along the depth of the model. The terms are defined in the glossary at the end, and each shows in the margin at its first use in a section.

In this grammar the op is not held at one token. The model infers it from the examples, and the anchor puts that evidence on e₁ at the example answers: an answer that difference could give comes onto e₁ in a context of any op (the example-evidence report found this). In a context of another op, about one example answer in 17 fitsFit Of an example or query answer: one that difference can give on its operands. difference. Most of these come from the replacement op noise, which drew difference for that example; in 13% the true op gives the same answer. Either way the model sees evidence for difference there, so turning those states is the removal at work rather than a side effect.

So the bound at the example answers counts only the answers that do not fit. Counting every answer would set it from the evidence instead: at slice 3 it would be 82°, against 22° without the fitting answers. The fitting answers get rows of their own in the E1 table.

How far the edit turns the other ops (E1)¶

We measure the turnWrite How far the edit turns a state: arcsin |α| of the alignment the state arrives with, taken at the 99th percentile over a group of states. At the embedding that is the clean α. At a later slice it is the α left after the block before has read the states the edit turned at the slices below. It is taken before the edit at that slice, which leaves no e₁ behind. on the contexts of the other ops, since that is where a turn is a side effect. On difference contexts the edit is meant to turn the states, and E2 and E3 score what that does.

The anti-subspace term is meant to keep the states of other ops off e₁, so we expect the edit to turn them only a little. We expect the turn to be largest at slice 1, where split-anti-seeds found the other ops closest to e₁ at the end of training. It should be smaller at slices 2 to 4, where they sat lower. Those are the slices the pull acts on, and they are also deeper; the control sits a little lower with depth too, so this study cannot say which matters.

Does the turn under the edit stay within the clean boundClean bound From the clean pass of the anchored model, before any edit: arcsin of the 99th percentile of |α| over the states of the six other ops at a slice and position. At the answers it counts only those that do not fit difference. It follows H4 of ex-2.2.1., at every slice and role? Or does a later block put e₁ back onto the other ops after the edit has taken it out at the slices before?

The figure shows the bound and the write at every position of a context, slice by slice, with the control and the difference contexts behind for reference. The two curves part most at the answers (y) and a little at =, and the write falls further below the bound with each slice.

The write stays within the boundWithin the bound The write is no larger than the bound over the same states. Ex-2.2.1 allowed a quarter over (a ratio of 1.25) as a tolerance; we quote the ratio. past the embedding. At the seed mean, the largest ratio of write to bound is 1.08, at slice 1, position ? of the third equation, where the bound is 0.3°. Run by run, 28 of the 768 sites past the embedding go over the tolerance of ex-2.2.1, all where the bound is at most 4°, and none by more than 1.7°.

At slice 1 the write is about the bound. Only the embedding has been edited before it, and the embedding holds little. From slice 2 on, the other ops arrive with much less e₁ than the clean model gives them. The only difference from the clean pass is the edits at the slices before, so the drop comes from those. In this model each block moves the state a quarter of the way toward the output of its attention and again toward its MLP, so about half of the incoming state carries through. On the clean pass that carried part brings its e₁ along; under the edit it has none, and what arrives is only what the block adds. At the example answers that do not fit, the write is 39% of the bound at slice 2 and 18% at slice 4.

That is below the control too, which has no anchor. From slice 2 on, the write at those answers is 11° to 4°, against 15° to 10° on the clean pass of the control. So the blocks after an edit do not put e₁ back on the other ops; they leave less there than either model has without the edit.

The bound is widest at the answers of the other ops. At the example answers it is widest at slice 1, as we expected: there the answers of the other ops come partway onto e₁, about twice as far as in the control. At the query answer it stays wide with depth, from 34° at slice 1 to 42° at slice 4, while at the example answers it narrows.

The wide bound at the query answer comes from a small tail. At slices 2 to 4, a few in a hundred contexts of other ops have their query answer close to e₁, while the median stays at a few degrees. How many varies a lot between seeds, from 61 contexts to 2,162. We found nothing in the examples that sets those contexts apart (their posterior on difference is about as low as in the rest), so this is unexplained.

The tail contexts are ones the clean model already answers less well: expected exact matchExpected exact match The task score, as the chance that a drawn answer agrees with a true one. With drawn answers, the correct answer to a line is spread over two or more colors, so we take the chance that an answer drawn from the model agrees with one drawn from the true distribution. Its ceiling is that same chance for the true distribution against itself (Σq²), below 1 on the ops that round. 0.47 against 0.54 in the rest, seed mean. Under the edit they end up further from the target nullTarget null Where a removal should land: what an ideal predictor that had lost one op and nothing else would answer. It is the Bayes posterior over ops given the examples, with the removed op set to zero and the rest renormalized, so it still weighs the other ops by how well they fit., but by about as much as they were already further from the Bayes predictorBayes predictor The best answer given the examples: it weighs the answers of each op by the posterior over ops, the probability of each op given the examples. on the clean pass, so the edit adds about the same distance there as elsewhere (0.012 against 0.010 in total variationTotal variation A distance between two answer distributions: half the summed absolute difference, from 0 (identical) to 1 (no overlap). Mass a model puts on tokens that are not colors counts as distance.). The edit takes these query answers off e₁ at the earlier slices, before they get there. At the other roles the bound is a few degrees at every slice. It is smaller than in the control at the colors, and about the same at the syntax.

The table puts the example answers in degrees. On the clean pass, the answers that fit difference climb onto e₁ with depth, to within a few degrees of it by slice 3. Under the edit they never get there. The edit at slice 1 turns them by about a quarter of a right angle, and at each later slice they arrive far less aligned than the clean model had them, so the writes after slice 1 are small.

Turning these states is the edit doing its job (see the Method): they are the evidence for difference in a context that otherwise points elsewhere, and E2 finds that removing them helps the contexts of other ops whose examples point at difference.

So on the recipe of record the bound holds in the sense of ex-2.2.1. The anchor gives a turn we can state in advance from the clean map, and the blocks downstream put back less than that.

The bound does not cover what the later blocks make of a state that was turned at slice 1; that is what the spillSpill The side effect of an edit on the other ops: the largest drop in the score of any op other than the one removed, net of what the same edit does to a control trained without the anchor. The selectivity criterion of D2.2 allows at most 0.02 at any dose. measures. At slice 1 the other ops are closer to e₁ than in a model with no anchor, so that is where any spill would start.

Whether the drop follows what each context needed (E2)¶

The target nullTarget null Where a removal should land: what an ideal predictor that had lost one op and nothing else would answer. It is the Bayes posterior over ops given the examples, with the removed op set to zero and the rest renormalized, so it still weighs the other ops by how well they fit. predicts the dropDrop The loss on one context under the edit: its expected exact match on the clean pass less that under the edit, net of the drop the same edit causes on the control at that context. context by context. A context whose examples all fit difference should lose most, and one whose examples point elsewhere should lose little. If the anchor holds the inferred op, the edit should take about the same share of the way to the target null in every group of contexts, whatever their posterior on difference. The posterior is what the examples say about the op, so an anchor that holds the posterior itself predicts the same; this test treats the two as one reading.

If the anchor holds something coarser, such as the label or a color feature of difference answers, the edit would take off about the same amount everywhere. That would be a larger share on the contexts that barely needed the op.

The main measure is the share removedShare removed Of a group of contexts: the mean drop over the mean dependence, so the part of the removal beyond what taking out an arbitrary axis costs. It is below the share without the control subtracted, since under the same edit the control loses more on difference contexts than on any other op (0.07 in expected exact match). by bin of the posterior. The correlation of drop with the dependenceDependence How much a context needed difference: the expected exact match of the clean model less that of the target null on the same context, so the most the edit could take off without going past the target null. of the Bayes predictorBayes predictor The best answer given the examples: it weighs the answers of each op by the posterior over ops, the probability of each op given the examples. is a second, weaker sign, since the clean model already tracks the posterior.

The target null also predicts a change on the contexts of other ops. Where an example showed the answer of difference, removing the op should move the answer back toward the true op, which is a gain.

The figure shows both: the share removed on difference contexts (left) and the gain on the other ops (right).

The share removed is about level across the posterior: between 0.75 and 0.88 at the seed mean, lowest in the 0.2–0.5 bin. The lowest bin is the noisiest, since its contexts needed little to begin with, and two seeds there go a little past the target null. No other bin shows the overshoot a coarse feature would give. The drop correlates with the Bayes dependence at r = 0.76 (0.69 to 0.83).

Without subtracting the control the share is level too, and higher: between 0.89 and 0.97 by bin. So the edit on its own takes difference contexts nearly all the way to the target null in expected exact matchExpected exact match The task score, as the chance that a drawn answer agrees with a true one. With drawn answers, the correct answer to a line is spread over two or more colors, so we take the chance that an answer drawn from the model agrees with one drawn from the true distribution. Its ceiling is that same chance for the true distribution against itself (Σq²), below 1 on the ops that round.. Part of that, 0.07 on average, is what the same edit costs a model with no anchor. The shape of the net share across bins comes from the control, whose own drop under the edit varies by bin, so that shape says where an arbitrary edit costs the most, rather than anything about the anchor.

On the contexts of other ops, the edit helps where their examples point at difference, and the more they point at it, the more it helps. That is the direction the target null predicts, and the edit goes most of the way. It falls furthest short where the examples are most in doubt between difference and the true op.

Where the posterior on difference is ≥ 0.5, the edit gains 0.23 against 0.33 for the target null. In the two low bins it gains a little more than the target null, 0.08 against 0.05 at 0.05–0.2.

Part of the excess in the low bins comes from subtracting the control, which loses a little there under the edit. Where the examples point most at difference the control gains a little instead, on every seed, so there the net gain is the smaller one. Without subtracting the control, the gains are 0.05 at 0.05–0.2 and 0.26 at ≥ 0.5. Most contexts of other ops have almost no posterior on difference (10,180 of 12,000 below 0.05), and there the target null predicts almost no change.

So the drop is graded the way claim (b) asks: in proportion to what each context needed. The gain on the other ops fits the anchor holding what the examples show about difference, wherever it appears, rather than what the label marks.

Where the edited answers fall short (E3)¶

On difference contexts the full edit covers about half of the distance from the clean answer distributions at the query to the target nullTarget null Where a removal should land: what an ideal predictor that had lost one op and nothing else would answer. It is the Bayes posterior over ops given the examples, with the removed op set to zero and the rest renormalized, so it still weighs the other ops by how well they fit.: a landingLanding How far the edit moves the answers toward the target null on difference contexts: the share of the clean distance to the target null that it covers, in total variation, as a ratio of the means over those contexts (the number split-anti-seeds quotes). of 0.52 (0.49 to 0.56). That is a different measure from the share removedShare removed Of a group of contexts: the mean drop over the mean dependence, so the part of the removal beyond what taking out an arbitrary axis costs. It is below the share without the control subtracted, since under the same edit the control loses more on difference contexts than on any other op (0.07 in expected exact match). in E2. E2 scores expected exact matchExpected exact match The task score, as the chance that a drawn answer agrees with a true one. With drawn answers, the correct answer to a line is spread over two or more colors, so we take the chance that an answer drawn from the model agrees with one drawn from the true distribution. Its ceiling is that same chance for the true distribution against itself (Σq²), below 1 on the ops that round., the probability on the right answer, which the edit takes most of the way to the null. The landing compares the whole answer distribution, including how the mass that leaves difference is shared among the other ops (the target null shares it by how well each op fits the examples).

Where is the rest? It could be on the answers of difference that the edit leaves behind, on colors no op gives, or on the answers of the other ops, weighed differently from the target null. These would point to different causes: a weak removal, damage to the answer, or a model that has lost difference and weighs what is left in its own way.

The edit moves nearly all the mass off the answers only difference gives, and onto the answers of the other ops. A little more lands on colors no op gives than on the clean pass. Nearly all of the distance leftDistance left The total variation distance between the edited answers and the target null, on difference contexts. is on the answers of the other ops.

Overall, the edited model spreads that mass over the other ops about as the target null does. Each op gets roughly its share, with lighten and hue-hsv a little short.

So the shortfall is context by context. It could be in how the edited model weighs the ops, or in how it shapes the answer of each op. To separate the two, we fit each edited answer distribution with the nearest mixture of the answer distributions of the ops, with non-negative weights.

We measure the two legs of the path through that mixture: from the edited answers to the mixture, and from the mixture to the target null. Total variationTotal variation A distance between two answer distributions: half the summed absolute difference, from 0 (identical) to 1 (no overlap). Mass a model puts on tokens that are not colors counts as distance. is a distance, so the two legs can add to more than the whole.

For reference, the clean model is some distance from the Bayes predictorBayes predictor The best answer given the examples: it weighs the answers of each op by the posterior over ops, the probability of each op given the examples. it learned to match. On difference contexts that distance is 0.16 (0.15 to 0.19), about as far as the control, and it is a little further on the other ops. This is a comparison point rather than a floor, since the two are distances between different pairs. But it says how close a model of this size gets to an ideal predictor when nothing has been edited. The figure puts that reference beside the distance left and the two legs.

The edited answers are close to a mixture of the answers of the ops (0.11 (0.10 to 0.11), against 0.04 on the clean pass). So they are mostly well formed: answers the other ops give, with about the right spread within each op.

The mixture, though, is 0.36 (0.33 to 0.38) from the target null, so the edited model weighs the other ops differently from the posterior, context by context. In all, the distance left is about 2.4× the reference.

The difference in weights is not a systematic one. Weighing the other ops more evenly than the target null, or more sharply, moves the prediction further from the edited answers: with the weights raised to a power of 0.5 or 2 and renormalized, the distance is 0.43 or 0.42, against 0.39 for the target null itself. With the overall proportions about right as well, what is left looks like scatter from context to context.

The edited model does use the examples to weigh the other ops, partly. Take a predictor that has lost difference and weighs the other ops equally, whatever the examples show. It is further from the edited answers (0.53) than the target null is (0.39).

Claim (c) asks that the edited model answer as the target null does. This fits the reading of it in the design doc, that the anchor gives the edit a direction and not the whole distance. The edit takes difference away cleanly, and the model answers with the other ops. It weighs them by the examples, but less accurately than the target null weighs them.

Discussion¶

On the recipe of record, the edit behaves the way the anchor says it should on all three claims, to a first approximation. The turnWrite How far the edit turns a state: arcsin |α| of the alignment the state arrives with, taken at the 99th percentile over a group of states. At the embedding that is the clean α. At a later slice it is the α left after the block before has read the states the edit turned at the slices below. It is taken before the edit at that slice, which leaves no e₁ behind. on the states of other ops is bounded by the clean map, slice by slice, and the blocks downstream of an edit leave less on e₁ than they found. The loss on each difference context follows how much that context needed the op, with no sign of an unintended, coarser feature.

The contexts of other ops whose examples point at difference recover as the target nullTarget null Where a removal should land: what an ideal predictor that had lost one op and nothing else would answer. It is the Bayes posterior over ops given the examples, with the removed op set to zero and the rest renormalized, so it still weighs the other ops by how well they fit. predicts. That fits the anchor holding the evidence for the op, wherever it appears, and the edit removing that evidence. In this grammar the op seems to be carried that way, spread over the example answers in the residual stream rather than held at one token. Whether part of it also lives in the attention, in what the heads match on (their query and key spaces) or what they carry (their value space), is open; we have put it in the backlog.

The shortfall is in where the answers land. The edit sends the mass to the right ops overall. But within a context, it shares the mass among them only partly by how well each fits the examples.

One reading is that the evidence the model uses to tell the other ops apart also travels partly along e₁, so taking out the axis blurs it. That would fit the anchor holding the evidence rather than the op itself. Another is that the model never needed fine weights among the other ops on difference contexts, since the answer there was difference, and so it never learned to weigh them there. This study does not separate them. A model trained without difference contexts would show how close any model of this size comes to the target null, and so how much of the shortfall is the edit; it is in the backlog.

If the spillSpill The side effect of an edit on the other ops: the largest drop in the score of any op other than the one removed, net of what the same edit does to a control trained without the anchor. The selectivity criterion of D2.2 allows at most 0.02 at any dose. grows on a later recipe, the boundClean bound From the clean pass of the anchored model, before any edit: arcsin of the 99th percentile of |α| over the states of the six other ops at a slice and position. At the answers it counts only those that do not fit difference. It follows H4 of ex-2.2.1. map at slice 1 is the first place to look, since that is where the bound is widest.

Glossary¶

Write

Turn

How far the edit turns a state: arcsin |α| of the alignment the state arrives with, taken at the 99th percentile over a group of states. At the embedding that is the clean α. At a later slice it is the α left after the block before has read the states the edit turned at the slices below. It is taken before the edit at that slice, which leaves no e₁ behind.

Clean bound

Bound

From the clean pass of the anchored model, before any edit: arcsin of the 99th percentile of |α| over the states of the six other ops at a slice and position. At the answers it counts only those that do not fit difference. It follows H4 of ex-2.2.1.

Within the bound

The write is no larger than the bound over the same states. Ex-2.2.1 allowed a quarter over (a ratio of 1.25) as a tolerance; we quote the ratio.

Fit

Of an example or query answer: one that difference can give on its operands.

Dependence

How much a context needed difference: the expected exact match of the clean model less that of the target null on the same context, so the most the edit could take off without going past the target null.

Drop

The loss on one context under the edit: its expected exact match on the clean pass less that under the edit, net of the drop the same edit causes on the control at that context.

Share removed

Net share

Of a group of contexts: the mean drop over the mean dependence, so the part of the removal beyond what taking out an arbitrary axis costs. It is below the share without the control subtracted, since under the same edit the control loses more on difference contexts than on any other op (0.07 in expected exact match).

Landing

How far the edit moves the answers toward the target null on difference contexts: the share of the clean distance to the target null that it covers, in total variation, as a ratio of the means over those contexts (the number split-anti-seeds quotes).

Distance left

The total variation distance between the edited answers and the target null, on difference contexts.