Where the spill of the edit comes from¶

The question comes from the backlog item, split from the withdrawn ex-2.2.24 draft. Ex-2.2.23 (E4) found that the edit spills onto other ops, mostly darken, lighten, and value-hsv, on nearly every run that has learned the HSV ops. This page applies the same edit to one set of positions at a time, on ex-2.2.23's checkpoints, with no new training.

Most anchored runs also have one syntax token whose embedding lies on e₁, a latchLatch A pooled pull that settles early on one cheap position and stays there, a different one from run to run: here, a syntax token whose embedding comes to lie on e₁ (ex-2.1.9). of the pooled pull. At 400 epochs that token is the separatorSeparator The , after each example answer, before the next example. on four of the twelve runs, and on those runs the separator is where most of the spill comes from.

Observations¶

• The sets of positions (E1): editing only the example answers removes difference as effectively as editing every position, and on most runs spills about as much, mostly onto darken at 400 epochs. The query positions do almost nothing. The rest of the positions spill without removing, mostly onto lighten.
• One position at a time (E2): each example answer alone takes the drop on difference about a fifth of the way, and spills a little. Across positions, the removal follows the alignment with e₁. No other single position removes anything, except the separatorsSeparator The , after each example answer, before the next example. on a few runs at 400 epochs.
• The latched separator (E3): those are the runs whose separator embedding lies on e₁ in every context, a latchLatch A pooled pull that settles early on one cheap position and stays there, a different one from run to run: here, a syntax token whose embedding comes to lie on e₁ (ex-2.1.9). of the pooled pull. Editing the first separator alone takes difference a third of the way or more and spills heavily. Editing only the example answers there spills a fraction of what the full edit does, though still more than the selectivity criterion of ex-2.2.21 allows.

Scope¶

This is an exploratory re-analysis, with no preregistration and no gate. It covers every run of ex-2.2.23: the recipe of record and the control, at 200 and 400 epochs, at model seeds 700 to 711. Every measurement of an anchored run is netted against the control at the same seed and length under the same edit, so what the edit does to a model without the anchor does not count as spill. The scores are on ex-2.2.21's held-out set of 3-example contexts, 2,000 per op.

The three anchored runs that ex-2.2.23's rule leaves out as half-trained (all at 200 epochs) are left out here too: they had not learned the HSV ops, and their edit hardly spilled. That leaves 9 runs at 200 epochs and 12 at 400. At 200 epochs most of those barely spill under any edit, and a few runs carry most of the spill, so the comparisons there rest on fewer runs than the count suggests. Runs differ a lot from seed to seed, so each table gives the seed range beside the seed mean, and the figures draw every run.

The measurements¶

The edit is ex-2.2.23's: at every slice, it removes a share (the dose) of the component of the state along e₁, the anchored direction, and rescales the state back onto the sphere. Here it acts on a chosen set of positions and leaves the others as they are. The answer is read at the query =, so only the 22 positions up to and including it can matter. The sets are:

• the example answers (y in each of the 3 examples);
• the query operands (a and b of the query);
• the query =;
• the rest: the example operands, every ? and example =, and the separatorsSeparator The , after each example answer, before the next example. (,) between examples.

Each set is edited at each dose, and so is every position together; each set is also left out of an otherwise full edit, at full dose; and each single position is edited alone, at full dose. Two measurements per edit, both from the expected exact matchExpected exact match The task score, as the chance that a drawn answer agrees with a true one. With drawn answers, the correct answer to a line is spread over two or more colors, so we take the chance that an answer drawn from the model agrees with one drawn from the true distribution. Its ceiling is that same chance for the true distribution against itself (Σq²), below 1 on the ops that round. on held-out contexts:

• Removal: how far the drop on difference gets toward the target nullTarget null What the model would answer without difference: the ideal predictor with that op removed from the posterior over ops, so it falls back on the ops that fit the examples next best., as a share of the way. Higher is more removal; 1 means the model answers as if difference were not among the ops.
• Spill: the largest drop on any other op. Lower is better; ex-2.2.21 (E2) set a selectivity criterion of 0.02, which we draw for reference.

Glossary¶

Target null

What the model would answer without difference: the ideal predictor with that op removed from the posterior over ops, so it falls back on the ops that fit the examples next best.

Separator

The , after each example answer, before the next example.

Latch

A pooled pull that settles early on one cheap position and stays there, a different one from run to run: here, a syntax token whose embedding comes to lie on e₁ (ex-2.1.9).

The sets of positions (E1)¶

Is there a set of positions where the edit removes difference without lowering other ops? Each set below is edited at four doses, and the figure plots the spill against the removal as the dose grows. A set that removes without spilling would run along the bottom of the panel to the right.

None does. The removal comes almost entirely from the example answers: editing them alone gets as far as editing every position, a little further on average. The spill comes with it. At 200 epochs the two lines lie nearly on top of each other, and at 400 epochs the example answers spill less than every position, though still far above the criterion on most runs.

The query positions do almost nothing, either way. The rest is the surprise: it spills without removing anything, as its line climbs straight up at 200 epochs. At 400 epochs it also removes on a few runs at full dose, from the separatorsSeparator The , after each example answer, before the next example. (E3).

The two kinds of spill land on different ops. Editing the example answers lowers darken most on most runs at 400 epochs, and editing the rest lowers lighten. Leaving the example answers out of the edit keeps the spill of the rest and loses nearly all of the removal at 200 epochs.

So the edit at the example answers removes difference and spills in the same act, and the other positions add a spill of their own. Restricting the edit to the example answers would drop the second kind and keep the first.

One position at a time (E2)¶

The sets in E1 lump positions of different roles together. Editing each position alone, at full dose, shows where within a set the removal and the spill sit, and whether the spill of the rest is spread over many positions or comes from a few.

Each example answer alone takes the drop on difference about a fifth of the way, and the three are alike. Together they take it further than their sum (0.91 at 400 epochs, against three times 0.22), so when one answer is edited, the other two make up part of the loss. No other position removes anything, except the separatorsSeparator The , after each example answer, before the next example. on the runs of E3.

The removal follows the alignment with e₁ on difference contexts, which ex-2.2.21 found highest at the example answers. Over every position of every run without a latched separator, the correlation between the alignment at a position (averaged over slices) and the removal when that position is edited is 0.93. On the runs of E3 it is lower (0.80), for the reason E3 gives.

The spill is spread thinly. Each example answer spills a little, and the operands, those of the query most, spill a little too. A handful pass the criterion on their own on most runs, and together they pass it by far: the spill of the rest in E1 comes from many positions with a small spill each, and from the separators on the runs of E3. So there is no single position to leave out of the edit that would take the spill away.

The latched separator (E3)¶

At 400 epochs, the separatorsSeparator The , after each example answer, before the next example. remove difference on some runs and not on others. Which runs, and why? Ex-2.2.21 (E1) noticed that the anchored runs put some syntax embeddings onto e₁, as seed means partway along it. Run by run, it is all or nothing: on most anchored runs the token embedding of one syntax token lies on e₁ (an alignment of 1 to two decimals), and the others stay near zero.

This is a latchLatch A pooled pull that settles early on one cheap position and stays there, a different one from run to run: here, a syntax token whose embedding comes to lie on e₁ (ex-2.1.9)., like the one ex-2.1.9 found under the pooled pull on the earlier grammar: the pull is pooled over the positions of a difference context at each slice, and at the embedding slice the cheapest way to meet it is to move one token that every context shares. The five runs where that token is the separator are the runs whose separators remove difference, four at 400 epochs and one at 200. On those runs the separator state lies on e₁ in every context, whatever the op, so it holds the token rather than the op.

Editing a latched separator removes nearly all of its embedding, so the blocks after it lose the separator in every context. On the runs at 400 epochs that removes difference a third of the way or more and spills nearly as much as the full edit. On the one run at 200 epochs it does much less, so the blocks there may lean on the separator less. Editing only the example answers removes as much as ever on these runs, and spills a fraction of what the full edit does.

The figure puts the two kinds of run side by side. Most runs sit on the diagonal: editing only the example answers changes the spill little. The latched runs at 400 epochs sit well above it, because their spill under the full edit comes mostly from the separators. Even there, the example-answer edit spills more than the criterion allows (from 0.04 to 0.09).

The other latches mostly fall outside this pass. The line break comes after the query, so its latch, the most common at 200 epochs, never reaches the answer here. The one run with a latched ? (seed 703, at 200 epochs) has the largest spill of the rest at that length, which fits the same account.

Discussion¶

The edit removes difference at the example answers, and on most runs that is where it spills too, by an amount that grows with the removal. That fits the spill being part of what e₁ holds at the example answers, beside the op, though it does not prove it.

What the anchor follows at the example answers found that the alignment there follows how well each example fits difference on its own. A judgement like that, made one example at a time, would also fire on examples of other ops whose answers happen to fit, and it may carry what makes an answer fit, such as how dark it is: difference answers are darker than their operands on average, and at 400 epochs the spill at the example answers lands mostly on darken. This pass does not separate those readings, since it keeps only the score per op.

On the runs without a latched separatorSeparator The , after each example answer, before the next example., the rest of the positions spill without removing, and at 400 epochs their spill lands mostly on lighten. Nothing at those positions helps the model infer the op, so whatever the edit takes from them is something other than the op. So at 400 epochs something besides the op may reach e₁ at the operands and symbols too, though the anti-subspace term is meant to keep it off.

If the edit were restricted to the example answers, it would lose the spill of the rest and of the separators, and keep the spill that comes with the removal. On most runs that leaves most of the spill. And restricting the edit is not a remedy that would work in natural language, where nobody marks the positions in advance. A selective edit needs e₁ to hold the anchored concept and nothing else, wherever in the context the anchor settles, which is a question about training that this pass can't answer.

A syntax embedding on e₁ is something other than the op, put there by the pull itself at the embedding slice, where a token that every context shares is the cheapest place to meet the pull. A softer pool would spread the pull over more positions, and a pull that leaves out the embedding slice kept every syntax embedding clean in ex-2.2.21 (no-emb); either could loosen the latchLatch A pooled pull that settles early on one cheap position and stays there, a different one from run to run: here, a syntax token whose embedding comes to lie on e₁ (ex-2.1.9)., though this pass only shows that the latch is there and what it costs the edit.