The pool temperature and anchor weight, without the embedding¶

Spill-by-position found that most anchored runs of the recipe latch one syntax embedding onto e₁, and that a latched separator carries much of the spill of the edit. The no-emb pull of ex-2.2.21, which leaves the embedding slice out of the anchor and anti-subspace terms, never latched there and scored best on the task, but only at three seeds and 200 epochs. The backlog item on a softer τ asked a related question: whether spreading the pull over more positions would move the anchor where it is needed.

This page trains no-emb at the recipe length over a log-log plane of τ and λ_a, one seed per trial, and fits smooth surfaces over the plane in place of repeated seeds. A few every-slice trials along τ are the reference.

Observations¶

• Removal along τ (E1): below τPool temperature The temperature of the mellowmax, a smooth stand-in for the maximum, that pools the alignment over the positions of a context: small τ is close to the largest alignment, large τ close to the mean. The recipe uses 0.1. ≈ 0.07 the edit barely removes difference on most trials, and from there up it removes most of the way on every trial. The weight makes little difference, and removal hardly changes past the edge.
• Where the anchor sits (E2): past the edge the anchor sits at the example answers. As the pool softens it spreads onto every =, and the syntax embeddings come partway onto e₁, with no full latchLatch A pooled pull that settles on one cheap position that every context shares, a different one from run to run: here, a syntax token whose embedding comes to lie on e₁ (spill-by-position, E3).. At the sharpest pool, four of the six trials put the = embedding most of the way onto e₁, a latch in a new place.
• Spill (E3): spill grows as the pool softens, together with the anchor at =. At the recipe τ, no-emb spills about as much as the every-slice runs that did not latch. The least spill, among trials that remove the op, is on the edge itself; the fits put the best point near τ = 0.07, λ_a = 0.13, though not with much confidence.

Scope¶

This is an exploratory study, with no preregistration and no gate. It trains 32 no-emb trials and 8 every-slice trials, each at its own model seed (800 to 839) and for 400 epochs, on the recipe of record: ex-2.2.21's corpus with 3 examples per context, the whole-line label, no cap. The no-emb trials are scattered over τPool temperature The temperature of the mellowmax, a smooth stand-in for the maximum, that pools the alignment over the positions of a context: small τ is close to the largest alignment, large τ close to the mean. The recipe uses 0.1. from 0.01 to 1 and λ_a from 0.025 to 0.4, both on log scales, by a scrambled Sobol sequence (a quasi-random pattern that covers a square more evenly than random draws). The every-slice trials are spaced evenly along τ, on a log scale, at the recipe weight of 0.1. Beside them are ex-2.2.23's 12 every-slice runs at 400 epochs, all at the recipe point (τ = λ_a = 0.1), which show how far runs differ by seed alone at one point.

The recipe sets the weight of the anti-subspace term as a multiple of λ_a, so the two terms move together and the ratio between them is the same on every trial. The sweep says nothing about the anti-subspace term relative to the pull.

With one seed per trial, how far a difference can be trusted comes from the fits (see below), which also estimate how much the trials scatter about the surface. At the recipe point, removal is nearly the same from seed to seed, and spill is not: its seed standard deviation is 0.13, as large as most of the differences in E3. So a single trial says little about spill, and a pattern has to show across several neighboring trials before we read anything into it.

The measurements¶

Each measurement is taken on ex-2.2.21's held-out set, as in ex-2.2.23. The edit is ex-2.2.22's: at every position and every slice, it removes a share (the dose) of the component of the state along e₁, the anchored direction.

• Removal: how far the drop on difference at full dose gets toward the target nullTarget null What the model would answer without difference: the ideal predictor with that op removed from the posterior over ops., as a share of the way. Higher is better.
• Spill: the largest drop on any other op, at any dose. Lower is better; ex-2.2.21 set a selectivity criterion of 0.02, which no trial here meets.
• α: the alignment with e₁ (the cosine between a state and the anchored direction), on difference contexts, averaged over slices 2 to 4: at the example answers, and at every = (the examples' and the query).
• Syntax embeddings: the largest alignment with e₁ of any syntax token embedding (?, =, ,, ⏎).
• Task cost: the seed mean of the controls minus the trial score.

Both edit measurements are net of the seed mean of ex-2.2.23's 12 controls at 400 epochs, so what the edit does to a model with no anchor is not counted. Ex-2.2.23 netted against the control at the same seed; here the trials have new seeds, and on ex-2.2.23's runs pairing by seed made little difference.

Each continuous measurement gets a Gaussian processGaussian process A smoother that fits a surface through scattered points and reports its own uncertainty at every point of the surface. Here it fits each measurement over log τ and log λ_a, with a separate term for how much single trials scatter about the surface. over the plane, fit in log τPool temperature The temperature of the mellowmax, a smooth stand-in for the maximum, that pools the alignment over the positions of a context: small τ is close to the largest alignment, large τ close to the mean. The recipe uses 0.1. and log λ_a on the no-emb trials, leaving out the four whose anchor sits on = (E2). The design also planned logistic fits for two yes/no outcomes, but neither varied: every trial made the second rise, the late jump in task skill when the model learns the three HSV ops (its worst HSV op ends above 0.3), and no no-emb trial latched a syntax embedding at spill-by-position's level of 0.9, though the = came close on four (E2). Task cost is small everywhere, at most 0.06, with no pattern over the plane; it is in the table at the end, and in E3 where it bears on the trials with the least spill.

Glossary¶

Pool temperature

τ

The temperature of the mellowmax, a smooth stand-in for the maximum, that pools the alignment over the positions of a context: small τ is close to the largest alignment, large τ close to the mean. The recipe uses 0.1.

Target null

What the model would answer without difference: the ideal predictor with that op removed from the posterior over ops.

Latch

A pooled pull that settles on one cheap position that every context shares, a different one from run to run: here, a syntax token whose embedding comes to lie on e₁ (spill-by-position, E3).

Gaussian process

A smoother that fits a surface through scattered points and reports its own uncertainty at every point of the surface. Here it fits each measurement over log τ and log λ_a, with a separate term for how much single trials scatter about the surface.

Removal along τ (E1)¶

Where on the plane does the edit remove difference? The figure plots removal against τPool temperature The temperature of the mellowmax, a smooth stand-in for the maximum, that pools the alignment over the positions of a context: small τ is close to the largest alignment, large τ close to the mean. The recipe uses 0.1., with the weight as shade, and the anchor at the example answers beside it, since that is where the example-evidence report found it.

Removal depends mostly on τ, through a sharp edge near τ = 0.07 (placed by eye). The fit rises again at the sharpest pool, but that rests on one trial. Above it, every no-emb trial removes the op most of the way, about as far as the every-slice runs at the recipe point, and softening the pool further changes little. Below it, most trials remove little. The every-slice trials follow the same edge, a little sharper.

The weight makes little difference. The fit finds no change with λ_a over the sixteenfold range of the sweep, and the shades mix on both sides of the edge. Just below it, the two trials that remove the op best are among the heavier ones, which is a hint at most.

The anchor at the example answers follows the same edge: below it, α there is low, and above it, α settles near 0.7 on every trial. So with a sharp pool, the pull is met somewhere the answer does not depend on, and the edit has nothing to remove. The trials at the sharpest pool that do remove the op are the exception, and E2 shows where their anchor went.

Where the anchor sits (E2)¶

Does the pool spread the anchor beyond the example answers as it softens, and does leaving out the embedding keep the syntax embeddings off e₁? The figure plots α at every = and the largest alignment of a syntax embedding.

At the sharpest pool, the anchor of four trials moves to =. On those trials the = embedding lies most of the way along e₁, even though the embedding slice is not pulled, and α at every = is high on difference contexts, and on the contexts of other ops too. The pull at the output of the first block is cheapest to meet by moving a token that every context shares, so leaving out the embedding slice moves the latchLatch A pooled pull that settles on one cheap position that every context shares, a different one from run to run: here, a syntax token whose embedding comes to lie on e₁ (spill-by-position, E3). one slice up rather than removing it. On those trials the edit removes the op by removing = everywhere (E3 shows what that costs). The other two trials at the sharpest pool have no anchor in use.

From the edge up to τPool temperature The temperature of the mellowmax, a smooth stand-in for the maximum, that pools the alignment over the positions of a context: small τ is close to the largest alignment, large τ close to the mean. The recipe uses 0.1. ≈ 0.2 the anchor stays at the example answers, and the syntax embeddings stay near zero, as in ex-2.2.21. As the pool softens beyond that, α rises at every =, and the most aligned syntax embedding comes partway onto e₁, usually the line break or the separator. None reaches a full latch, and the weight makes little difference.

The every-slice pull latches at half the values of τ, much as it did at the recipe point: four of its 8 trials have one syntax embedding on e₁, and at the softest pool its syntax embeddings rise partway too.

Spill (E3)¶

How much does the edit spill across the plane, and does it follow where the anchor sits? The figure plots spill along τPool temperature The temperature of the mellowmax, a smooth stand-in for the maximum, that pools the alignment over the positions of a context: small τ is close to the largest alignment, large τ close to the mean. The recipe uses 0.1., as in E1.

Past the edge, spill grows as the pool softens. Near the recipe τ (0.08 to 0.2), the no-emb trials spill about as much as the every-slice runs at the recipe point that did not latchLatch A pooled pull that settles on one cheap position that every context shares, a different one from run to run: here, a syntax token whose embedding comes to lie on e₁ (spill-by-position, E3)., and less than the ones that did. So leaving out the embedding slice takes away the extra spill of the latch, and leaves the rest. The four trials with the anchor on = spill most of all, since their edit removes = in every context. Below the edge, the trials spill little, and remove little.

The spill follows the anchor at =. Among the trials that remove the op, leaving out the four on =, the correlation between spill and α at every = is 0.69.

The trials that spread the anchor onto = spill most, and those that keep it at the example answers spill least, though even those spill well above the criterion of ex-2.2.21. The every-slice runs at the recipe point have little α at = and spill as widely as the no-emb trials, so = is one source of spill among others.

Putting removal and spill together, the most selective trials tend to sit near the edge, where the pool is just soft enough for the anchor to settle at the example answers and no softer. The three trials with the least spill among those that remove at least 0.7 of the way are no-emb-04 (τ = 0.08, λ_a = 0.39), no-emb-06 (τ = 0.19, λ_a = 0.10), and no-emb-20 (τ = 0.07, λ_a = 0.18). Two of them are among the few trials that cost the task something (no-emb-04 and no-emb-06, whose worst HSV op ends lower than on most trials), so a model that learned the task less well may account for part of their low spill. The third has no task cost, and it sits on the edge.

On the fitted surfaces, the point of least spill where removal reaches 0.7 is at τ = 0.07 and λ_a = 0.13. There the fits give removal 0.7 ± 0.1 and spill 0.1 ± 0.1. The edge is narrow along τ and the spill is noisy, so the region is a thin band beside a steep slope, rather than a broad plateau.

Discussion¶

The pull kept off the embedding slice does what spill-by-position hoped for over most of the plane: no syntax embedding latchesLatch A pooled pull that settles on one cheap position that every context shares, a different one from run to run: here, a syntax token whose embedding comes to lie on e₁ (spill-by-position, E3)., and the extra spill that a latched separator brings goes away. But the edit still spills at the recipe τPool temperature The temperature of the mellowmax, a smooth stand-in for the maximum, that pools the alignment over the positions of a context: small τ is close to the largest alignment, large τ close to the mean. The recipe uses 0.1., about as much as on the every-slice runs without a latch. So most of the spill on a fully trained model has some other source. A later re-analysis, where the lean comes from, traces it on the every-slice runs to lightness loaded onto e₁ in the color embedding table, and finds the no-emb trials here leaning the same way. The no-emb runs of ex-2.2.21 and ex-2.2.22 spilled little, and at 200 epochs on three seeds, that may have been the short training more than the pull, though this sweep has no 200-epoch arm to separate the two.

Leaving out the embedding slice also moves the latch rather than removing it, when the pool is sharp. At the sharpest τ the pull is met at the first slice it reaches, by putting the = embedding on e₁. That fits the account of the latch in spill-by-position, a pool that is close to a max is met most cheaply by a position every context shares, and it suggests that leaving out more slices might only move the latch further up at a sharp enough pool, though only the embedding slice was left out here.

Softening the pool spreads the anchor, as the backlog item expected, but onto = and the syntax tokens, and the spill grows with it. The anchor does reach the query =, though only together with every other =. So a softer pool does not help the edit here: by the time the anchor reaches the query, the spill has grown.

So within this plane, the least spill comes at the edge where the anchor first settles at the example answers. It is a narrow band beside a steep slope, and the noise in the spill is about as large as the differences along it, so we hold the proposed point loosely: with one trial per point, we cannot tell whether the spill drops on the edge or the trials there were lucky draws. Two of the three least-spilling trials also cost the task something (E3), which leaves one clean trial on the edge. If the spill does drop there, removal and spill are traded along τ, and λ_a barely enters removal; the spill fit varies with λ_a only through single trials. Since λ_a scales the anti-subspace term too, the ratio between the two terms is the same on every trial, so the sweep says nothing about a heavier anti-subspace term relative to the pull.

All trials¶