Ex 2.2.21: the in-context grammar pilot, on the reworked recipe¶

A second try at round 2 of the D2.2 route, on the recipe that ex-2.2.17 to ex-2.2.20 reworked. We retrain the control and the arms that anchor difference under the whole-line label and its variants, then score the rules that ex-2.2.16 never reached, to decide how round 3 anchors and edits the inferred op.

Findings¶

• The recipe holds, and the anchor costs nothing (H1) — partial. The control reproduces ex-2.2.19. The whole-line arm falls short of the control by more than the tolerance, but by less than the seeds vary.
• The label rule (S1) — decided: the whole-line label stays. No variant beats it by more than the seeds vary.
• Where the anchor sits (E1) — on the answers, where the context shows the most evidence about its op, and hardly at all at the query =.
• Editing the op out (E2) — on the hinge arm, an edit at every position takes the op out gradually and leaves the other ops alone. On the whole-line arm the same edit spills just past the gate onto another op. No edit at the query = meets both criteria on any arm.
• The verification rule (S2) — decided: verification lines stay, since they leave completion unchanged.
• The leans stay with the control (H2) — pass. The anchor adds no lean toward itself.

How to read this draft¶

The rules and predictions were frozen at commit 2d84d2b, before any run of this experiment. The preview scores the stored runs of ex-2.2.16; it predates this plan and shaped it.

Why this experiment¶

Ex-2.2.16 was round 2 of the D2.2 route. It trained a control and several arms that anchor difference on the in-context grammar, and froze rules to select how round 3 would anchor and edit: the label, the site, the operator, and whether verification lines stay in the corpus. Round 3 would then anchor the inferred op at fresh seeds and suppress it.

But ex-2.2.16 stopped at its first rule. Its control had learned less than half of what the examples allow, 39% of the way from the floor to the Bayes ceiling. An anchor compared with a model that far from finished would say little about round 3.

Four reports then worked on the control alone: ex-2.2.17 added the newline mask, a lower peak learning rate, and more steps; ex-2.2.18 dropped four ops whose answers often coincide with another op; ex-2.2.19 settled the length at 200 epochs; and ex-2.2.20 kept the plain schedule.

On the seven-op set the control now scores 0.554 against a ceiling of 0.609, 88% of the way from the floor. This pilot reruns round 2 on that recipe.

It keeps the label, verification, and operator rules of ex-2.2.16, changing each only where the stored runs of ex-2.2.16 or the new recipe call for it, and adds three label variants as references. The design names the rules (a) to (e); here they are H1, S1, and S2, and the site and operator rules become an exploratory analysis, E2.

What ex-2.2.16's anchored arms already show¶

Ex-2.2.16's anchored arms were trained and evaluated, but the eval was never scored. These numbers come from the half-trained model, so they only show what to look for.

It turns out the anchor landed. The op marginOp margin How far the contexts of the anchored op sit along e₁ beyond the rest. At each slice, take the mean alignment over difference contexts less the mean over all contexts, difference included, at the role where that gap is largest; then average over slices. The anchor term optimizes this quantity, so it checks that the pull landed., which measures how far difference contexts sit along the anchored axis beyond the rest, was 0.32 on the whole-line arm against 0.01 on the control. But most of it landed where an edit cannot reach the answer:

On the whole-line arm the alignmentAlignment The cosine between a state and e₁, the anchored axis, per position and slice. Cosine similarity compares direction only: 1 when two vectors point the same way, 0 when they are perpendicular. (the cosine of a state with e₁) sits on the answers. It builds up on the example answers by block 2 (0.36 to 0.51) and fades a little after; on the query answer it climbs with depth, to 0.91 at the last block. The query =, whose state predicts the answer, stays below 0.07 at every slice.

The pooled anchor term asks a labelled context to align somewhere in its span. The answers are where the context has shown the most evidence about its op, so the pull seems to have settled there.

For an edit, the two kinds of answer differ. The state at the query answer comes after the answer is predicted (the model reads left to right) and predicts only the line break, so an edit there cannot change the answer. The query can read the example answers through the blocks above them, so an edit there may still reach it, though not an edit at the last slice, which no block reads.

The query ? stayed clean (0.00). Ex-2.2.16 worried that it would saturate, but the position that came close was the query answer. The hinge arm, which caps the pull, holds 0.28 at the query = at the last block, but the other ops hold nearly as much there (0.21), so that alignment says little about difference.

The suppression pass agrees:

At the query =, an edit on the whole-line arm barely touches the answer, and on the hinge arm it lowers every op about as much as difference.

With the edit at every position and every slice, the projection on the whole-line arm grows with dose up to γ = 0.75, then levels off 34% of the way to the target nullTarget null The answer a model that has lost difference and nothing else would give: the posterior-weighted answer distribution with difference removed and the other ops renormalized.. From γ = 0.75 on, it also spills past the selectivity gate onto other ops. The hinge arm gets 66% of the way and spills more.

At the embedding slice the two arms differ. There the solid and dashed traces coincide, because a token embedding is the same whatever the op. Each anchored arm has put a few syntax embeddings partway onto e₁:

• on the whole-line arm, ? (0.32), , (0.34), and the line break;
• on the hinge arm, = (0.27) and ,.

On the earlier grammar a leak like this came through the tied readout (the embedding table reused as the readout table), and untying it brought the leak down (ex-2.2.7). The readout here is already untied. So this leak seems to come from the pull itself: the pull acts on the embedding slice of every position it pulls, and some of those are tokens every context shares.

On the whole-line arm the bump at , sits one position after the answers. That makes the alignment look as though it moves back a token with depth, but the two are separate effects.

On the hinge arm the = embedding explains why the query = holds alignment on every op. It starts at 0.27 for every op and keeps about that much to the last block, while the example = positions fade to 0.05 to 0.15. An edit at every position moves these syntax states in every context, which may be part of why it spills onto other ops.

The preview changes four things in this plan:

• Three new label variants move the pull toward the query =, as references rather than candidates for round 3. prompt leaves the query answer out of the pull, to see where the pooled term settles without it. query-eq pulls the query = alone: a position oracle,¹1 The term from ex-2.1.8: a pull at a position chosen by hand, which no labeller could give. It is a reference for what the right position would allow, and not a strict ceiling. which shows what an anchor there would allow. every-eq pulls every =, the positions whose next token depends on the op, to see whether a pull spread over the examples settles at the query = too.
• The saturation half of ex-2.2.16's hinge rule goes, since the query ? did not saturate, though the hinge arm stays a candidate.
• The site and operator rules become an exploratory analysis that chooses no arm. The suppression pass also edits the example answers, and the edits are scored at the query = as well as at every position.
• E1 reports the syntax embeddings beside the per-position alignment, and the no-emb arm, which leaves the embedding slice out of the pull, shows whether they stay clean without it.

Glossary¶

Terms follow ex-2.2.16; these are the ones the rules here use.

Alignment

The cosine between a state and e₁, the anchored axis, per position and slice. Cosine similarity compares direction only: 1 when two vectors point the same way, 0 when they are perpendicular.

Op margin

How far the contexts of the anchored op sit along e₁ beyond the rest. At each slice, take the mean alignment over difference contexts less the mean over all contexts, difference included, at the role where that gap is largest; then average over slices. The anchor term optimizes this quantity, so it checks that the pull landed.

Expected exact match (EEM)

The probability mass the model puts on the colors the query answer can be under the true op, weighted by how often the op gives each, averaged over held-out contexts. The task measurement.

Bayes ceiling and floor

The expected exact match of a predictor that holds the posterior over ops given the examples, and of one that ignores the examples.

Target null

The answer a model that has lost difference and nothing else would give: the posterior-weighted answer distribution with difference removed and the other ops renormalized.

Seed band

The smallest difference between two seed means a comparison resolves: 2σ√(1/n₁ + 1/n₂) for arms at n₁ and n₂ seeds, with σ the seed standard deviation pooled over the two. At the spread of ex-2.2.19's four runs it is about 0.008 for five seeds against five, 0.009 for five against three, and 0.010 for three against three.

Conditions¶

Every arm trains at the center condition of ex-2.2.16 (three examples, ρ = 0.3) on the seven-op set, under ex-2.2.19's recipe: the newline mask, a peak rate of 0.00316, 200 epochs, d64-L4. The anchored arms add ex-2.2.14's anchor recipe, each weight scheduled as a fraction of training as before, and crop policy whole.

Ex-2.2.16's arms at other corpus conditions, its larger control, and its two newline-mask arms are gone: the condition is settled, the larger control was for a shortfall the recipe has since closed, and the mask is now in every arm.

The new label variants. All three keep ex-2.2.16's labeller: a difference context draws a label with probability 0.02, keyed on the op array beside the corpus. Variant (e), prompt, pulls every position of a labelled context up to and including the query =, leaving out the query answer and the line break. Variant (f), query-eq, pulls the query = alone. Variant (g), every-eq, pulls the = of each example and of the query, the four positions that predict an answer. The first example's = comes before any evidence, so its pull asks for the op before the context shows it.

prompt keeps the example answers in the pull, because they are part of the evidence. If the pull still settles on them, that is a result about pooling.

The seeds. Fresh model seeds from 700. The control and the whole-line arm get 5 each, since every comparison runs through one of them; the others get three.

The recipe holds, and the anchor costs nothing (H1)¶ ~ Partial

What we expect. Round 3 starts from this recipe only if both of these criteria hold:

• (a) The control reproduces ex-2.2.19: its seed-mean held-out expected exact matchExpected exact match (EEM) The probability mass the model puts on the colors the query answer can be under the true op, weighted by how often the op gives each, averaged over held-out contexts. The task measurement. is within 0.015 of 0.554, the mean of ex-2.2.19's four runs at 200 epochs.
• (b) The whole-line arm falls short of the control by at most 0.01, a little wider than the seed bandSeed band The smallest difference between two seed means a comparison resolves: 2σ√(1/n₁ + 1/n₂) for arms at n₁ and n₂ seeds, with σ the seed standard deviation pooled over the two. At the spread of ex-2.2.19's four runs it is about 0.008 for five seeds against five, 0.009 for five against three, and 0.010 for three against three. of five seeds against five.

The arms differ from ex-2.2.19 only in their seeds and in the anchor, so (a) checks that nothing else moved.

We expect (b) to hold too. Ex-2.2.14 found the anchor costs the task nothing when 2% of contexts are labelled, and the stored runs of ex-2.2.16 agree: the whole-line arm scored 0.272 against 0.266 for the control, and 0.311 against 0.308 with the newline mask.

A miss on (b) would mean the anchor and the task compete on this recipe, and would leave E2 with no candidate, since it asks the same of each.

This replaces corpus rule (a) of ex-2.2.16, which asked the control to come within 0.03 of the calibrated ceiling. The seven-op control misses that: it sits 0.055 below the ceiling at 200 epochs, and still misses at 400. Ex-2.2.17 found the remaining gap on contexts whose examples settle the op, where the model keeps mass on the answers of a similar op.

We take the control as it stands. Its skill score and calibration KL²2 A KL divergence: a non-negative measure, in nats, of how far one probability distribution sits from another, 0 when they match. Here it compares the model's answer distribution with the Bayes predictor's. are reported beside every comparison, but no criterion depends on them. In ex-2.2.19 the KL was about 0.32 nats; ex-2.2.16 called a model calibrated below 0.05.

What we saw. The control scores 0.5435, 0.0102 from ex-2.2.19's 0.5537, so (a) holds. The whole-line arm scores 0.5116, 0.0319 short of the control against a tolerance of 0.01, so (b) misses. That shortfall is inside the seed band of the pair, 0.059: the control has a seed standard deviation of 0.031, against 0.006 over ex-2.2.19's four runs.

The skill score is 0.86 for the control and 0.79 for the whole-line arm, and the calibration KL is 0.35 and 0.43 nats (0.32 in ex-2.2.19).

The seeds also differ in how fast they learn. This figure was drawn after the results came in, to look at the spread over seeds:

Model seed 702 has the lowest held-out EEM in 8 of the 12 arms, and the lowest run of every arm is still rising over the last 20 epochs, by 0.003 to 0.012.

The label rule (S1)¶ → Decided

The rule. Ex-2.2.16's rule (b), unchanged. The whole-line label stays the primary unless a candidate variant (below) beats the whole-line arm on seed-mean held-out expected exact matchExpected exact match (EEM) The probability mass the model puts on the colors the query answer can be under the true op, weighted by how often the op gives each, averaged over held-out contexts. The task measurement. by more than the seed bandSeed band The smallest difference between two seed means a comparison resolves: 2σ√(1/n₁ + 1/n₂) for arms at n₁ and n₂ seeds, with σ the seed standard deviation pooled over the two. At the spread of ex-2.2.19's four runs it is about 0.008 for five seeds against five, 0.009 for five against three, and 0.010 for three against three., while keeping an op marginOp margin How far the contexts of the anchored op sit along e₁ beyond the rest. At each slice, take the mean alignment over difference contexts less the mean over all contexts, difference included, at the role where that gap is largest; then average over slices. The anchor term optimizes this quantity, so it checks that the pull landed. of at least 90% of the whole-line margin. If several qualify, the one with the higher op margin goes forward.

The candidates are latter, prefix, and no-emb (the whole-line label with the embedding slice left out of the pull). Variant (d), sampled, trains the grading that round 3 will measure, so it is reported and not promoted.

What we expect. No variant clears the gate, for the reasons ex-2.2.16 gave: the label share barely moved the margin in ex-2.2.14, and the variants change less than that.

What we saw. No candidate qualifies. Each scores above the whole-line arm, by less than the seed band of the pair: no-emb by +0.050 against 0.070, latter by +0.033 against 0.074, and prefix by +0.035 against 0.073. no-emb keeps 1.02 of the whole-line op margin, prefix 0.85, and latter 0.76, below the 90% floor. sampled, not a candidate, scores +0.012 against the whole-line arm and keeps 1.00 of its margin.

Where the anchor sits (E1)¶

Where an edit can work depends on where along the context each anchored arm puts its alignmentAlignment The cosine between a state and e₁, the anchored axis, per position and slice. Cosine similarity compares direction only: 1 when two vectors point the same way, 0 when they are perpendicular.. E1 measures, per anchored arm, the seed-mean alignment at each position and slice of a held-out context, on difference contexts and on the other ops.

What we saw. At the last block the query = holds 0.02 to 0.07 on difference contexts on every arm but two: query-eq at 0.82 and every-eq at 0.80, where the other ops hold 0.60 and 0.44 at the same position. The whole-line arm holds 0.97 at the query answer and 0.68 on the example answers; prompt, whose pull leaves out the query answer, holds 0.03 and 0.73. The query ? stays within 0.02 of zero on every arm.

At the embedding slice the line-break embedding sits at 0.80 to 1.00 on whole-line, latter, prefix, sampled, hinge. no-emb keeps every syntax embedding within 0.12 of zero. prompt lifts , to 0.67 and ? to 0.35, and query-eq and every-eq lift = to 0.96 and 0.96.

Editing the op out (E2)¶

E2 asks whether difference can be edited out of an anchored model with a dial: an edit whose effect on difference grows with its dose while the other ops stay as they were. It chooses no arm, so round 3 is designed after this report, and it replaces hinge rule (c) and operator rule (e) of ex-2.2.16.

The candidates (anchor-whole, anchor-hinge) are arms round 3 could adopt as they stand. The references (anchor-prompt, anchor-query-eq, anchor-every-eq) put the pull where no labeller could.

The edits. The suppression pass of ex-2.2.16 runs, scoring only, on the candidates, the references, and the control. It has three operators, each applied at every slice at once:

• the projection, which removes a share γ of the component along e₁, for γ in {0.25, 0.5, 0.75, 1};
• the repulsion, which moves states aligned above 0.5 down to an alignmentAlignment The cosine between a state and e₁, the anchored axis, per position and slice. Cosine similarity compares direction only: 1 when two vectors point the same way, 0 when they are perpendicular. of 0.25 and then 0;
• the reflection, which flips the component, at one dose.

Each edit is applied at one of four sites: the query ?, the query =, the example answers, or every position. Every drop listed below is a net drop: the seed-mean fall in held-out expected exact matchExpected exact match (EEM) The probability mass the model puts on the colors the query answer can be under the true op, weighted by how often the op gives each, averaged over held-out contexts. The task measurement., minus the fall the control shows under the same edit, so only the part the anchor caused counts.

The two criteria. Two sites are scored against the criteria below: the query =, and every position (the edit that would carry over most readily to another task). At the other two sites the drops are reported without criteria. An operator at a scored site meets the criteria on an arm when both of these hold:

• It grades. The net drop on difference contexts rises with the dose, allowing a dip between adjacent doses of at most 0.01, and at full dose it covers at least 50% of the distance from the clean score to the target nullTarget null The answer a model that has lost difference and nothing else would give: the posterior-weighted answer distribution with difference removed and the other ops renormalized.. The reflection has one dose, so it cannot grade.
• It is selective. On each of the other six ops, the net drop is at most 0.02 at every dose. This is the threshold ex-2.2.11 set for a change in expected exact match too small to matter for the task. It sits well above the seed bandSeed band The smallest difference between two seed means a comparison resolves: 2σ√(1/n₁ + 1/n₂) for arms at n₁ and n₂ seeds, with σ the seed standard deviation pooled over the two. At the spread of ex-2.2.19's four runs it is about 0.008 for five seeds against five, 0.009 for five against three, and 0.010 for three against three. of three seeds against three, because the worst of six ops is the largest of six noisy numbers.

A flat threshold treats every other op alike, but some are easier to mistake for difference than others. So beside it, E2 shows the net drop on each other op against how plausible difference is in that op's contexts: the mean posterior on difference given the examples. If the drop rises with that posterior, the edit takes out the op as the model infers it, and the cost on other ops follows the confusions the contexts allow.

Where the op comes from. The bypass test in the design for round 3 edits at different positions and slices, looking for a route by which the op reaches the answer around the edit. E2 is a first look at it, varying the position only. If the edit at every position works and the one at the query = does not, the answer must take the op from somewhere else, and the edit at the example answers shows whether that is the examples.

What we expect. Neither candidate meets both criteria. On the whole-line arm the projection at every position levels off about a third of the way to the null, as in the preview, and on the hinge arm the edit at every position spills onto other ops.

On query-eq we expect the projection at the query = to meet both, which would say that an anchor there can be edited with a dial, if a label could put it there. On every-eq we expect the same, more weakly, since a quarter of its pull lands on the query =. We are unsure about prompt: its pull may settle on the example answers, which E1 will show.

What we saw. Of the 30 operators, sites, and arms scored, 2 meet both criteria, both on the hinge arm at every position: the projection, which covers 0.17, 0.54, 0.76, 0.81 of the way to the target null over its four doses with the worst other op at most 0.002, and the repulsion, which covers 0.51, 0.61 with the worst other op at most -0.004. On the whole-line arm the projection at every position grades the same way (0.18, 0.55, 0.76, 0.81), and the worst other op reaches 0.026 at full dose, over the gate of 0.02.

At the query =, no edit on the whole-line, hinge, or prompt arm covers more than 0.00 of the way to the null. On query-eq and every-eq the projection there grades, to 0.91 and 0.93 at full dose, and the worst other op reaches 0.59 and 0.63. On prompt the projection at every position grades, with the worst other op at 0.27.

At the example answers, the reported site, the full projection covers whole-line 0.91, hinge 0.91, prompt 0.58, query-eq 0.00, every-eq -0.01 of the way to the null. So on the whole-line and hinge arms, where the edit at every position works and the one at the query = does not, editing the examples alone takes most of the op out: the query seems to take the op from the examples.

The drop on each other op against the posterior (preregistered)

The figure plots the drop on each other op under the full projection against the mean posterior on difference in the contexts of that op. Over the six other ops that posterior runs only from 0.038 to 0.040, so the ops bunch on the x-axis and the figure cannot show whether the drop follows it. The next section asks the same question another way.

Post hoc: what the model answers instead¶

The net drop says how much of difference an edit takes out, not what the model answers in its place. A pass over the same runs, added after the results were in, measures that: for each held-out context, the probability the model puts on the answer of each op, on the clean model and under the full projection at every position. Beside it is the Bayes predictor with difference removed from the posterior, which is what a model should answer once it no longer knows the op.

On the whole-line and hinge arms, the edited model answers difference contexts much as that predictor does. The mass on the difference answer falls from 0.71 to 0.06 on the whole-line arm and from 0.71 to 0.06 on the hinge arm, and moves to the other ops in about the proportions of the predictor.

The other contexts hardly change: their mass on the true op moves from 0.48 to 0.45 on the whole-line arm and from 0.48 to 0.46 on the hinge arm, against 0.52 to 0.48 on the control. So on these arms the edit seems to remove the op from the inference and leave the rest in place.

On query-eq and every-eq the same edit takes out every op: no cell of either matrix holds more than 0.13, so the model stops giving an answer that any op would give. prompt sits between: its difference row spreads like the predictor, and the other contexts keep 0.25 on the true op, against 0.51 clean.

Post hoc: editing the no-emb arm¶

no-emb had the highest task score of any arm in S1 and kept the anchor. After the results were in, we added its three runs to the suppression pass, to see whether its anchor edits as well as the candidates. Nothing here was predicted or given a criterion in advance; the figure applies the E2 criteria for comparison.

The edit takes difference out of no-emb faster than out of the candidates, and takes more of the other ops with it. At every position the projection covers 0.22, 0.70, 0.84, 0.84 of the way to the target null over its four doses, against 0.18, 0.55, 0.76, 0.81 on the whole-line arm. It grades, but the worst other op reaches 0.077 under the projection and 0.081 under the repulsion, several times the gate of 0.02.

The edit stays selective up to the half dose, where the projection covers 0.70 of the way to the null. On the candidates, a selective projection gets as far as 0.76 on the whole-line arm (γ 0.75) and 0.81 on the hinge arm (γ 1). So leaving out the embedding slice buys no edit the candidates lack.

In the matrices above, no-emb answers difference contexts much as the whole-line arm does once edited. Most of its spill is on darken contexts, whose mass on their own answer falls from 0.71 to 0.58, against 0.71 to 0.65 on the control.

The E1 alignment suggests why. At the example answers in blocks 1 and 2, the other ops sit higher on e₁ on no-emb (0.37, 0.27) than on the whole-line arm (0.26, 0.15), so an edit there removes more of what the other ops use. Leaving the embedding slice out of the pull seems to make the early blocks hold the axis less selectively.

The verification rule (S2)¶ → Decided

The rule. Ex-2.2.16's rule (d), unchanged. Verification lines stay in the corpus from round 3 on if they leave completion unchanged: the seed-mean held-out expected exact matchExpected exact match (EEM) The probability mass the model puts on the colors the query answer can be under the true op, weighted by how often the op gives each, averaged over held-out contexts. The task measurement. on completion contexts is within the seed bandSeed band The smallest difference between two seed means a comparison resolves: 2σ√(1/n₁ + 1/n₂) for arms at n₁ and n₂ seeds, with σ the seed standard deviation pooled over the two. At the spread of ex-2.2.19's four runs it is about 0.008 for five seeds against five, 0.009 for five against three, and 0.010 for three against three. of the arm without them, on both pairs (control-verify against control, anchor-verify against anchor-whole). The verification accuracy and the op marginOp margin How far the contexts of the anchored op sit along e₁ beyond the rest. At each slice, take the mean alignment over difference contexts less the mean over all contexts, difference included, at the role where that gap is largest; then average over slices. The anchor term optimizes this quantity, so it checks that the pull landed. of anchor-verify are reported with no gate.

What we expect. Completion is unchanged on both pairs. The verification arms see 70% of the completion lines the others do. On the old recipe that cost the control nothing measurable; at 200 epochs each line is seen more often, so the cost should be smaller still.

What we saw. Both pairs are within their seed band. control-verify scores -0.036 against the control, within 0.073, and anchor-verify scores +0.022 against the whole-line arm, within 0.072. Verification accuracy is 0.84 and 0.85, and the op margin of anchor-verify is 0.501, against 0.481 for the whole-line arm.

The leans stay with the control (H2)¶ ✓ Pass

What we expect. On the whole-line arm, the first-operand lean and the trailing-fragment lean stay within the seed bandSeed band The smallest difference between two seed means a comparison resolves: 2σ√(1/n₁ + 1/n₂) for arms at n₁ and n₂ seeds, with σ the seed standard deviation pooled over the two. At the spread of ex-2.2.19's four runs it is about 0.008 for five seeds against five, 0.009 for five against three, and 0.010 for three against three. of the control, as ex-2.2.16 defined them. The newline mask now keeps a context from reading the one before it, which should make a cut-off fragment look less like a whole context than it did in ex-2.2.15.

What we saw. On the whole-line arm the first-operand lean is 0.020 against 0.066 for the control, a difference of -0.046 within the seed band of 0.053. The trailing-fragment lean is 0.035 against 0.028, +0.007 within 0.031. Among the other anchored arms, compared with the control at their own seed band, only query-eq and every-eq fall outside, both below the control on the first-operand lean.

Discussion¶

Most of the recipe goes forward to round 3: the control reproduces ex-2.2.19, and the whole-line label and the verification lines stay. Whether the anchor costs the task anything stays open, since the shortfall misses the tolerance but sits inside the seed bandSeed band The smallest difference between two seed means a comparison resolves: 2σ√(1/n₁ + 1/n₂) for arms at n₁ and n₂ seeds, with σ the seed standard deviation pooled over the two. At the spread of ex-2.2.19's four runs it is about 0.008 for five seeds against five, 0.009 for five against three, and 0.010 for three against three..

One seed lags on most arms and is still climbing at the last epoch, so five seeds at 200 epochs can't tell a small cost from a slow start. A longer schedule would let the slow seeds finish, and is the likeliest way to settle it.

On the whole-line and hinge arms the anchor sits on the example answers. These come before the query =, where the answer is predicted, so the op seems to be anchored upstream of the point we score. The edit works when applied everywhere, examples included; an edit at the query = alone has little to act on.

Where the edit works, the post hoc answer matrices suggest that the model answers as if it no longer knew the op but still weighed the other ops.

The arms that pull the anchor onto the query = also lift the = embedding, which every op shares. That may be why their edits spill onto the other ops.

An edit at every position on the hinge arm is the clearest lead for round 3. On the whole-line arm, whose pull is uncapped, the edit grades as well, but it spills just over the gate. So a cap on the alignmentAlignment The cosine between a state and e₁, the anchored axis, per position and slice. Cosine similarity compares direction only: 1 when two vectors point the same way, 0 when they are perpendicular. in [0.8, 1), where 1 would be no cap, may keep the selectivity with more of the alignment.

The no-emb arm is the open puzzle: it had the highest task score of any arm, but its anchor edits less selectively. Perhaps leaving the embedding slice out of the pull makes the early blocks overcompensate, holding the axis for the other ops too. Or the anchor weight was set with the embedding in the pull, and no-emb would edit more selectively at a lower weight. This experiment can't separate the two.

Method¶

The corpus and the posterior¶

The corpus is ex-2.2.18's seven-op corpus at the center condition, which ex-2.2.19 also trained on: one context per line, 300,000 contexts, three examples and ρ = 0.3, cube noise at κ = 0.02, and stochastic rounding. The held-out and probe sets are ex-2.2.18's. The verification arms train on the same generator with 30% of contexts written as verification lines.

Dropping four ops leaves the posterior on difference much as it was: on the seven-op table 42% of difference contexts sit in the middle band, against 42% on eleven ops, with 24% below it and 34% above. On these sampled contexts the Bayes ceiling is 0.613 and the floor 0.149; on the held-out set of ex-2.2.19, the ceiling is 0.609.

But the posterior takes few distinct values: 89% of the contexts in the middle band sit in one step between 0.85 and 0.95, so the stimulus has about three levels. That matters for the graded stimulus of round 3, but not for the rules here.

The measurements¶

Ex-2.2.16's measurements, scored on its held-out sets at the end of training, with the mask on in every arm: expected exact matchExpected exact match (EEM) The probability mass the model puts on the colors the query answer can be under the true op, weighted by how often the op gives each, averaged over held-out contexts. The task measurement. against the ceiling and floor, the calibration KL, the alignmentAlignment The cosine between a state and e₁, the anchored axis, per position and slice. Cosine similarity compares direction only: 1 when two vectors point the same way, 0 when they are perpendicular. by position and slice, the alignment against the posterior, the op marginOp margin How far the contexts of the anchored op sit along e₁ beyond the rest. At each slice, take the mean alignment over difference contexts less the mean over all contexts, difference included, at the role where that gap is largest; then average over slices. The anchor term optimizes this quantity, so it checks that the pull landed., the two leans, the suppression pass, and verification accuracy. The op margin and the leans are also recorded through training on a fixed probe set. Comparisons are between seed means, with the seed bandSeed band The smallest difference between two seed means a comparison resolves: 2σ√(1/n₁ + 1/n₂) for arms at n₁ and n₂ seeds, with σ the seed standard deviation pooled over the two. At the spread of ex-2.2.19's four runs it is about 0.008 for five seeds against five, 0.009 for five against three, and 0.010 for three against three. as the resolution; the seed standard deviation is pooled over the two arms compared.

The verification arms are scored for expected exact match, the op margin, the alignment, and the leans on the main held-out set (the same contexts as every other arm), and for verification accuracy on the verification held-out set.

The new variants¶

prompt, query-eq, and every-eq are masks, by role within the context, on the positions a labelled context pulls (roles as listed under Conditions). They run through the same pooled term as every other variant, which gives each labelled context the same total pull however many positions share it.

On query-eq the pool has one position, so the whole pull lands on the query =: the variant changes how hard that position is pulled as well as where. On every-eq the pool has four positions, so the query = gets a quarter of that.

E2 scores query-eq and every-eq without separating the two changes; E1 shows the alignment it reaches at the query = beside the other arms.

Budget¶

40 runs at about $0.13 each on an L4, from ex-2.2.19's unanchored runs at 200 epochs; we expect the anchored step to cost about the same and will check on the first runs. The suppression pass is scoring only, on 22 checkpoints. Under $10 in all.