Anchoring on an axis the optimizer does not favor¶

On stored checkpoints, we found that every run parks its separators (?, , and ⏎) one-hot on a coordinate of their own and = on a second, and that a coordinate axis is cheaper to project out than a random direction. In short training runs, Adam built this basis, and Adam in a rotated basis built the same thing on the rotated axes.

The anchoring design had taken e₁ for an arbitrary direction. If it is not, the results that rest on e₁ might lean on it being a cheap axis: the anchor itself, or the comparisons around it, or both. This study trains sixteen new runs, with the basis of the optimizer rotated and nothing else changed: falloff (the recipe of record of split-anti-seeds) and the un-anchored control, at the eight seeds where both are stored.

Observations¶

• Does the anchor hold? (E1): yes, on every seed. The alignmentAlignment How far a state points along e₁: the cosine between the state and e₁, which for a unit-length state is its component along e₁; 1 on the anchor direction and 0 off it. at the example answers rises with depth as in the stored runs, and the profiles of the new and stored runs lie on top of each other.
• Removal, spill and landing (E2)Spill The side effect of an edit on the other ops: the largest drop in the score of any op other than the one removed, net of what the same edit does to a control trained without the anchor. The selectivity criterion of D2.2 allows at most 0.02 at any dose.: on every seed the edit removes nearly all of the op and spills within the criterion, as before. The removal net of the control falls short on 2 seeds, and all of the shortfall comes from the control: where the rotation puts e₁ near the syntax direction of the control, the edit removes part of the op from the control as well.
• Where the privileged basis went (E3): to the rotated axes. The energy spectrum is flat along the coordinate axes and spiked along one rotated axis, which holds the syntax tokens. In a control, e₁ costs what an ordinary direction costs, and more where it overlaps that spike.

Scope and method¶

This is an exploratory study, with no preregistration and no gate. It covers falloff (the recipe of record) and the control at model seeds 704 to 711, trained for 400 epochs with Adam's per-coordinate scaling applied in a random orthogonal rotation of the residual coordinates (one rotation per seed, drawn from the model seed).

Each new run pairs with a stored twin at the same seed (the falloff run from split-anti-seeds or the 400-epoch control from ex-2.2.23). The pair shares its initialization, batches and label draws, and differs only in the basis of the optimizer. The model, schedules, anchoring code, measurements and dose scoring are those of split-anti-seeds, unchanged.

A difference between a new run and its twin is compared with the seed range of the stored arm. Split-anti-seeds found falloff within 0.002 of split-anti on removal, with every seed within the spillSpill The side effect of an edit on the other ops: the largest drop in the score of any op other than the one removed, net of what the same edit does to a control trained without the anchor. The selectivity criterion of D2.2 allows at most 0.02 at any dose. criterion.

The loss-rate screen put the loss rate near one in forty, so one loss in eight here would be unremarkable. Its upper bound is about one in nine, which leaves two losses in eight inside chance as well. So a change in the loss rate would show only at three or more losses; a smaller effect would show in αAlignment How far a state points along e₁: the cosine between the state and e₁, which for a unit-length state is its component along e₁; 1 on the anchor direction and 0 off it. on the pulled slices, or in removal on the seeds that kept the anchor.

The measurements¶

• Whether the anchor held: αAlignment How far a state points along e₁: the cosine between the state and e₁, which for a unit-length state is its component along e₁; 1 on the anchor direction and 0 off it. at the example answers at the end of training, per slice, measured as in split-anti-seeds. A run that lost the anchor sits near zero on the pulled slices. It counts as lost when the full edit removes less than half of difference, net of the control (the rule of split-anti-seeds and the loss-rate screen).
• Removal, spillSpill The side effect of an edit on the other ops: the largest drop in the score of any op other than the one removed, net of what the same edit does to a control trained without the anchor. The selectivity criterion of D2.2 allows at most 0.02 at any dose., and landing: the dose scoring of ex-2.2.22 (the projection at every slice and position, at four doses), net of the control at the same seed. Removal is the net drop on difference over its clean-to-null span, spill the largest net drop on another op, and landing the fraction of the distance to the target nullTarget null Where a removal should land: what an ideal predictor that had lost one op and nothing else would answer. It is the Bayes posterior over ops given the examples, with the removed op set to zero and the rest renormalized, so it still weighs the other ops by how well they fit. that the full edit covers. E2 also shows removal without subtracting the control, since the control turns out to matter here.
• The task scores of every op before any edit, compared with those of the control.
• The basis measurement, new here: the share of the energy of the stream¹1 Energy: the sum of the squared projections of the states onto a direction. The share of a direction is its energy over the total. that each direction holds, along the coordinate axes and along the rotated basis, and the mean kurtosis²2 Kurtosis: how heavy the tails of a distribution are. A Gaussian scores 3; a direction where most values are small and a few are large scores higher. along each. It also records the cost of projecting out one direction at full dose, for e₁, each coordinate axis, each rotated axis, and sixteen random directions. The stored runs give the reference, from our check on their checkpoints: one coordinate holds about half of the energy, a typical axis costs 0.03, and a random direction 0.08 to 0.13.

Does the anchor hold? (E1)¶

With the basis rotated, is e₁ still held through training at every seed, and as firmly? If the anchor leans on e₁ being a privileged axis, some seeds should lose it, or αAlignment How far a state points along e₁: the cosine between the state and e₁, which for a unit-length state is its component along e₁; 1 on the anchor direction and 0 off it. at the example answers should sit lower on the pulled slices than in the stored falloff runs. If not, the per-slice α profiles should lie within the stored seed range.

A second explanation needs keeping apart. In the stored runs the syntax spike sits on a coordinate axis, orthogonal to e₁ by construction; under the rotation it sits on a rotated axis whose cosine with e₁ is a few tenths at most, and known per seed.

So an anchor that is harder to hold could mean that the anchor needs a privileged axis, or that the anti term now has to push the syntax spike off e₁. E3 reports that cosine per seed, to show how much of any seed-to-seed difference it accounts for.

What we saw. The anchor held on every seed. The alignment profile of each new run lies on that of its stored twin at every slice after the embedding (the largest gap on any seed and slice is 0.04), and the eight new profiles span the same range the eight stored ones do.

The one visible difference is at the embedding, where the new runs sit at about 0.06 against 0.02 in the stored runs. The anti weight is highest there, and under the rotation the syntax tokens no longer have a coordinate of their own, orthogonal to e₁ for free. So the few hundredths may be what it costs the anti term to keep them off e₁.

But the same uplift shows in the contexts of the other ops, which fits another explanation at least as well: under the rotation e₁ is a generic direction at the embedding, so every embedding has a small share of it. Past the embedding, the alignment in those contexts is unchanged.

So the anchor does not need e₁ to be a privileged axis: the pull finds and holds an ordinary direction as well as it held an axis.

Removal, spill and landing (E2)¶

On the seeds that kept the anchor, does the edit remove difference as it did, and does it spillSpill The side effect of an edit on the other ops: the largest drop in the score of any op other than the one removed, net of what the same edit does to a control trained without the anchor. The selectivity criterion of D2.2 allows at most 0.02 at any dose. as little? The stored falloff runs removed four fifths to nine tenths of the op at full dose with every seed within the spill criterion of 0.02. The comparison is paired by seed, against the new control at the same seed.

What we saw. Two things happened at once, and the table separates them. First, the anchored runs themselves remove the op as they did: before subtracting the control, the drop on difference under the full edit is 0.98 (0.97 to 0.99) of the clean-to-null span on the new runs and 0.96 (0.87 to 0.99) on the stored ones.

Every new run spills within the criterion (the largest net spill over the doses is 0.000 (−0.009 to 0.005), and the raw spill at the full dose, 0.017 (−0.002 to 0.038), matches what the stored runs show). The landing is 0.50 (0.42 to 0.54) against 0.52 (0.49 to 0.56) stored, and the task change is within the seed range (−0.001 (−0.013 to 0.006)).

Second, the removal net of the control (the measurement the earlier reports quote) falls short on 2 seeds by the kept/lost rule, and is lower on a third. The shortfall sits in the control. In the stored runs the control loses about a tenth of the op when e₁ is projected out, because e₁ there is an ordinary coordinate axis that holds little.

Under the rotation e₁ is an ordinary direction instead, which is not orthogonal to the syntax spike of the model. On seeds 706 and 710 the cosine between e₁ and the spike direction of the control is about a quarter, and the control loses half or more of the op to the edit on its own. The shortfall in net removal tracks that cosine across the eight seeds (Pearson r = 0.94).

So the kept/lost rule, applied as written, would count 2 of 8 new runs as having lost the anchor. That is inside chance by the reckoning in Scope, and the raw removal and the alignmentAlignment How far a state points along e₁: the cosine between the state and e₁, which for a unit-length state is its component along e₁; 1 on the anchor direction and 0 off it. both say that no run lost the anchor (0 are below half by the raw drop).

The drops on the other ops show the same thing from the other side. On seeds 706 and 710 the control loses on every op, up to 0.31 (0.18 to 0.44) in score on an op other than difference, while the anchored run at the same seeds loses 0.024 (0.022 to 0.025) at most on any other op. The stored controls lose at most 0.10 on any other op.

So the edit on the rotated control takes syntax, spread over the whole task, while the edit on the anchored run takes the anchored concept and little else. Netting against the control assumes the edit on the control touches nothing much, which held when e₁ was a quiet axis and does not hold for a direction that overlaps the spike of the control.

At those seeds the anchored run holds its own syntax spike on a rotated axis away from e₁ (E3), which fits the anti term steering it there. So e₁ is clean in the anchored run where it is not in the control.

Where the privileged basis went (E3)¶

Did the rotation do what it was meant to? If so, the energy spectrum should be flat along the coordinate axes and spiked along the rotated axes, with the syntax tokens on a rotated axis. And e₁ should cost what an ordinary direction costs in the stored runs (0.08 to 0.13 in a control, where it touches the spikes), not what an ordinary axis costs (0.03).

In the anchored runs e₁ holds the anchored op, so its cost there is the removal plus the spillSpill The side effect of an edit on the other ops: the largest drop in the score of any op other than the one removed, net of what the same edit does to a control trained without the anchor. The selectivity criterion of D2.2 allows at most 0.02 at any dose., and it is compared with e₁ in the stored falloff runs. The cleaner comparison is within a run (e₁ against the random directions of that run and its rotated axes off the spikes), since the stored reference numbers are means over draws that touch the spikes by varying amounts.

The basis measurement also records where each token embedding sits (its largest cosine with a coordinate axis and with a rotated axis), so the published data show whether the syntax tokens moved to a rotated axis.

What we saw. The privileged basis moved with the optimizer. Along the rotated basis one direction holds 0.48 (0.46 to 0.50) of the energy of the stream at the last slice, as the one coordinate did in the stored runs. Along the coordinate axes the largest share is 0.07 (0.05 to 0.10), and the spectrum is much flatter, though it still spans about an order of magnitude from the largest direction to the smallest.

Kurtosis runs the reverse of the stored runs: near the Gaussian value of 3 along the axes (3.2 (2.9 to 3.4) in the controls), and above it along the rotated basis (4.2 (4.0 to 4.4)).

The syntax tokens sit on that rotated spike: the ?, ,, ⏎ and = embeddings have a cosine of 0.88 (0.74 to 0.99) with a rotated basis vector, and on most seeds all four share the vector that holds the spike (the table). Against the coordinate axes their largest cosine is about a third, what a random unit vector in 64 dimensions gives.

On 3 of 8 seeds (704, 706, 710) the spike of the anchored run sits on a different rotated axis from that of the control, and on the others on the same one. In the stored runs we found the spike on the same coordinate in the control and the anchored run at every seed we checked. So a move here fits the anti term steering the choice, but the two runs also differ in their loss terms, and could settle differently for that reason alone.

e₁ now costs what a direction costs, plus the cost of its overlap with the spike. In the controls, the 63 other coordinate axes cost 0.07 at the median, about what a random direction costs in the stored controls, and a rotated axis off the spike costs 0.03, what an ordinary axis costs in the stored runs. So the cheap directions are now the rotated axes.

In the controls, e₁ costs 0.05 (0.03 to 0.09) on the six seeds where its cosine with the spike is about a tenth or less, and 0.19 (0.17 to 0.22) on the two where it is about a quarter, the overlap E2 found in the net removal. In the falloff runs e₁ costs 0.10 (0.09 to 0.10) on every seed: the removal of the anchored op, which the cost spreads over the ops, with nothing of the spike in it.

Discussion¶

The anchor did not need a privileged axis. Pulled toward an ordinary direction, falloff aligned, removed and stayed selective as it did on the coordinate axis on all eight seeds, apart from a few hundredths of alignmentAlignment How far a state points along e₁: the cosine between the state and e₁, which for a unit-length state is its component along e₁; 1 on the anchor direction and 0 off it. at the embedding. So the assumption of the design, that e₁ stands in for any direction, holds for the anchor itself. The privileged basis changes how we should interpret the comparisons around the anchor, not what the anchor does.

What the rotation did change is the control, which the removal and spillSpill The side effect of an edit on the other ops: the largest drop in the score of any op other than the one removed, net of what the same edit does to a control trained without the anchor. The selectivity criterion of D2.2 allows at most 0.02 at any dose. measurements are netted against. When e₁ was an axis, the syntax tokens of the control sat on another axis, orthogonal to e₁ for free (in the anchored runs of earlier designs the anchor pulled a syntax token onto e₁ instead, the latch of spill-by-position). When e₁ is a direction, it overlaps the spike of the control by a random cosine, and on two of eight seeds that was enough for the edit to take half the op from the control, so the net measurement counts a loss the anchored run never had. Where the edit damages every op of the control, as on those two seeds, subtracting it no longer isolates the anchored op.

So the baseline needs choosing with the basis in mind, whatever the optimizer. One that does not depend on where e₁ happens to sit is the median drop over edits along several other directions, like the reference edit of the spill-timing experiment: coordinate axes off the two syntax coordinates under Adam, and any directions under an optimizer with no privileged basis. Raw removal beside it shows what the anchored run itself gives up.

A reader of an earlier report need not revise anything, since every stored comparison kept e₁ as an axis in both runs. But the same logic applies to reference directions: a reference direction compared with e₁ should be another coordinate axis off the two syntax coordinates, since a random direction overstates the baseline by whatever it takes from the spike.

The anchored runs also show the anti term at work in a way the stored runs could not. With the coordinates privileged, keeping the syntax off e₁ cost nothing, because the model parked it on a different coordinate anyway. Here, on 3 seeds, the anchored run holds its syntax on a different rotated axis from the one the control chose.

That fits the anti term steering the choice of the model among the cheap directions of the optimizer. But one of the three is a seed where e₁ overlaps nothing, and the two runs may have settled differently for reasons of their own.

If this holds, the anti term would have a larger job still in a model with no privileged basis at all, since there would be no cheap direction to steer the syntax onto. The optimizer scout shows what the stream of such a model looks like without an anchor.