Ex 2.2.16: the in-context grammar pilot¶

This is round 2 of the quick route. The method section folds in the posterior scouting report: it computes the posterior over ops and the Bayes ceiling from the op table, with no training, and uses them to pick the three corpus conditions for the control.

Findings¶

Each rule was to propose a setting for round 3. Only the first was scored.

• The corpus rule (a): no condition passes. The d64-L4 control scored 0.27 at the center against a pass line of 0.49, the other two conditions and the d128-L4 control fell as far short, and none is calibrated. Under the frozen rule, this is the outcome that adds a round.
• Rules (b) to (e) and the two predictions: not reached. They compare anchored arms with a control that has learned about half of what its examples allow, so the comparison would mix the effect of the anchor with unfinished training.

How to read this report¶

This was a preregistration. The conditions, the five rules with their thresholds, and the two predictions were frozen at commit 49e59eb, before any run; later edits to them are immaterial or are marked here. The method section on the posterior and the ceiling was computed before training.

Every arm trained and was evaluated, and the eval is in the store, but only rule (a) is scored here. The sections for the other rules keep their frozen text, each with a note that it was not reached. Terms are defined where they first appear and collected in the glossary.

Why this experiment¶

Ex-2.2.14 anchored an op and every gate passed, but the anchor went to the word that names the op, so the anchored concept was an attribute of one token. The pivot takes op words out of the grammar.

Each line is one context: a few solved examples of one op, written with ? in place of the op word, then a query under the same op. The op is inferred from the examples, and the anchored concept is "the op in this context is difference", which no token names.

This pilot scouts the posterior (see Method below), trains the new-grammar control, smoke-tests anchoring the inferred op, and tries the label variants.

The control is the regression check; a grammar this different needs its own, as ex-2.2.3 did. No model trained on the corpus can know more about the op than the examples say, and the posterior captures that. So instead of fixed accuracy numbers, the task gate is how far the control falls short of the ceiling the posterior sets, with an analytic floor beside it.

The crop policy starts as whole, which pulls only labelled lines wholly inside the training window. Ex-2.2.15 found that lines cut short by the window show the first-operand lean (a leak toward the anchor). Its review chose whole over half, which its rule had set, because a whole line always shows its evidence, which matters more once the evidence is a set of examples, and its discussion asks the pilot to watch the trailing-fragment lean as well.

Glossary¶

Context

One line of the corpus: k examples of one op, then a query. Its op is inferred from the examples.

Example

A solved equation inside a context, a ? b = y,: the evidence the op is inferred from.

Query

The last equation of a context, whose answer the model completes. It is always clean.

Replacement op noise

Showing, in some examples, the answer another op would give, at a rate ρ per example. It spreads the posterior over ops, so it grades the stimulus.

Cube noise

Showing, in some examples, a color drawn from the whole grid, at a rate κ. Usually no op produces it, so it removes the evidence of an example without pointing anywhere else.

Posterior on the true op

How strongly the examples of a context point to the op that generated it, computed from the op table under the stochastic rounding and replacement op noise the corpus uses. For difference contexts it is the graded stimulus.

Bayes ceiling

The expected exact match on the query of the predictor that holds the posterior over ops and answers with the posterior-weighted answer distribution: the score of a calibrated model that has learned everything the examples say, which is where cross-entropy training aims.

Floor

The same predictor with no evidence: uniform over ops, so its answer distribution is the op-marginal. It is what a model that ignores the examples can score.

Skill score

Where a score sits between the floor (0) and the ceiling (1): (score − floor) / (ceiling − floor).

Expected exact match (EEM)

The probability mass the model puts on the colors the answer of the query can be under the true op, weighted by how often the op gives each, per context, then averaged over held-out contexts. The task metric of every report since ex-2.2.9; the ceiling and the floor are in the same units.

KL divergence

How far one distribution is from another, in nats: KL(q ‖ p) = Σ_y q(y) log(q(y) / p(y)), zero when they agree. Here q is the Bayes predictive on a query and p the model's answer distribution, and the mean over contexts is the model's excess loss over a calibrated one, so 0.05 nats is a twentieth of a nat of loss the model could still shed. The calibration check gives reference values.

Alignment

The cosine between a state and e₁, the anchored axis: 1 when the state points along the axis, 0 when perpendicular. Measured per position and slice.

Lean (ᾱ)

The mean alignment at a role over the held-out contexts of every op: a leak toward the anchor at a site that carries no evidence about the op. The first-operand lean is ᾱ at the first token of a context, at the last block; the trailing-fragment lean is ᾱ over every position, at the last block, of a context cut to start partway through, fed on its own. Both from ex-2.2.15.

Op margin

How far the contexts of the anchored op sit along e₁ beyond the rest: per slice, the mean alignment over the contexts of the anchored op minus the mean over the contexts of all eleven ops, at the role where that gap is largest, averaged over slices. The quantity the anchor term optimizes, so it checks that the pull landed. The statistic of ex-2.2.14, with roles running over the whole context.

Seed band

The smallest difference between two seed means that a comparison resolves: 2σ√(2/n) at n seeds per arm, with σ the seed standard deviation pooled over the two arms. Rules (b) and (d) use it on expected exact match at the center condition.

Conditions¶

The pilot trains 14 arms at 3 seeds each, 42 runs. The control runs at the three corpus conditions the method section proposes, k3-r0.2, k3-r0.3, k4-r0.3 (three or four examples per context, at a replacement rate of 0.2 or 0.3), at d64-L4, and once more at d128-L4 at the center. The larger control is wider rather than deeper: the part of the task we are least sure a small model can do is the lookup of each example against eleven ops, which we expect to need width more than depth, and keeping four blocks keeps the slice axis the same for every arm. It trains either way, so that a miss under rule (a) does not cost a round, and is scored only if rule (a) needs it. Everything else runs at the center condition, k3-r0.3.

The anchored arms all anchor difference on e₁ with the recipe from ex-2.2.14 and crop policy whole, and differ in what the label covers: the whole context, or one of the four label variants. A hinge arm caps the whole-line pull at an alignment of 0.8, so no state is asked to be all concept. Two pairs add something to the corpus or the model on both the control and the whole-line arm: verification lines, and the newline mask, under which attention does not cross a line break.

Ex-2.2.15 also carried an oracle arm, knowable, which pulled a whole labelled line only when its op was in view. Under whole every pulled context is wholly in view, so on this grammar that arm is the same as the whole-line arm, and the pilot leaves it out.

The labeller. A context of difference draws a label with probability 0.02, keyed on the op array beside the corpus rather than on any token; no context of another op is labelled. Under the whole-line label every token of a labelled context is pulled through the pooled anchor term, which asks the context to align somewhere in its span. Variant (a) leaves the embedding slice out of the pull; (b) pulls the latter half of the context; (c) pulls a position once the posterior on difference given the examples before it reaches 0.5; (d) makes the draw depend on the evidence: a context of difference is labelled with a probability proportional to its posterior on difference, with the constant set so that, on average, 2% of them are labelled, as in the other arms. A context whose examples barely fit difference is then rarely labelled, and one whose examples all but name it is labelled a little more often than in the other arms.

The seeds. Three per arm, all fresh; no arm is served from the store. Comparisons between arms are between seed means, with the seed band as the resolution.

The corpus rule (a)¶

The rule. A condition passes when the seed-mean held-out expected exact match of the d64-L4 control is at least the ceiling of record less 0.03. Among the passing conditions, the one with the widest spread of the posterior on the true op (its standard deviation across contexts, from the method section; the middle-band share breaks a tie) goes forward. Spread is the criterion because it is a property of the corpus: it says how graded the stimulus is, and round 3 measures the anchor against that grading. The calibration KL (below) is a property of the model, and would favor the condition with the least noise, where there is least to infer and least grading. So the KL is reported beside each condition, and a control that passes without being calibrated is flagged. If none passes, the d128-L4 control at the center is scored on the same line, and goes forward with the center condition if it passes; if it also falls short, the grammar is reworked before anything is anchored. That is the one outcome that adds a round.

The margin is one-sided. The calibrated ceiling is not the most a model can score: a model that sharpens past calibration can reach the hard-EM ceiling, which is higher (the method section has both). So a control above the ceiling passes, and the calibration check below says whether it got there by sharpening, which is worth knowing before round 3 reads its answer distribution. The bounds of record at each condition, and the score the control has to reach:

At the center the margin is about 7% of the room between floor and ceiling. For scale, a predictor that ignores one of the three examples and is otherwise Bayes-optimal gives up 0.097 there, so a control that passes has learned to use every example, near enough.

What we expect. The d64-L4 control passes at every condition, and k3-r0.3 goes forward. The pivot expects the model to be enough because there are no variables to track; the inference is a lookup over eleven ops per example and a pool over examples. A control that passes at k3-r0.2 and misses at the center would say the noise, not the grammar, is what costs it, and the fallback within the same block size is already in the table.

The calibration check. The check is the mean over held-out contexts of the KL divergence from the Bayes predictive \(q\) (the posterior-weighted answer distribution on the query) to the model's answer distribution \(p\) at the query =, in nats. Averaged over contexts, it is the model's cross-entropy on the answer less the Bayes-optimal cross-entropy: the loss the model could still shed. A control is calibrated at a condition when it is under 0.05 nats. The method section gives the scale: a tenth of the floor mixed into the Bayes answer costs about 0.043 at the center, ignoring one example 0.29, and committing to the most probable op 0.73. We expect the control to be calibrated at every condition it passes; a control that passes and is not calibrated has sharpened, and the direction of the miss (the KL grows fastest where the model puts no mass on an answer \(q\) allows, so a sharpened model shows it more than a hedging one) is reported beside it.

The result. No condition passes, and the d128-L4 control misses too:

The shortfall is large and much the same everywhere. Every control has a skill score between 0.38 and 0.46, a little under halfway from the floor to the ceiling, and every seed sits far below its pass line. Width did not help: the d128-L4 control at the center scored 0.291, against 0.266 for the d64-L4 control, a difference inside the spread of its own seeds (0.235 to 0.321). The calibration KL is 0.75 to 1.08 nats across the seeds, many times the threshold of 0.05.

So the grammar is reworked before anything is anchored, as the rule says. The miss alone does not say whether the grammar or the training recipe is the limit; the discussion has what came next.

The label rule (b)¶

The rule. The whole-line label stays the primary unless a variant clears the task gate: its seed-mean held-out expected exact match at the center, over all contexts, exceeds that of the whole-line arm by more than the seed band, while it holds the anchor: its op margin is at least 90% of the margin of the whole-line arm. If more than one variant qualifies, the one with the higher op margin goes forward. Variant (d) trains the grading that the evidence section reads, so it is reported on the same statistics and not promoted.

The op margin is the one ex-2.2.14 used, with the contexts of the anchored op as the labelled group and roles running over the whole context (the glossary has it). The variants narrow the pull, so their margins can only be lower by construction where the narrowed positions carried it; the bar asks that most of it survives.

What we expect. No variant clears the gate. Ex-2.2.14 found that the label share barely moves the margin and that the task is untouched at the rate of the primary, and the variants change less than that. What they may change is where the alignment sits: (a) should leave the embeddings clean, (b) and (c) should put less alignment on the first example, and (c) leaves the middle-band contexts unlabelled. Those are read in the alignment figures of the evidence section; this rule scores the task and the margin only.

Not reached. Rule (a) did not pass, so this is not scored. The arms trained, and their eval is in the store.

The hinge rule (c)¶

The rule. The uncapped whole-line arm saturates at the query ? when the seed-mean alignment at that position, on the held-out contexts of difference at the last block, is at least 0.9. If it does, the hinge arm goes forward in its place, provided the hinge arm holds the anchor (90% of the whole-line margin) and it is within the seed band of the whole-line arm on the task. If the whole-line arm does not saturate, it stays, and the hinge arm is reported.

Saturation is the dose collapse ex-2.2.14 found at the op word: a state that is all concept has no partial dose, because projecting e₁ out of it leaves a remainder too small to mean anything. The query ? is a constant token with nothing else to hold, so the pooled pull may put the alignment of the whole context there. The hinge counts an alignment of 0.8 as full: in the anchor term only, a position pays 1 − cos/0.8 in place of 1 − cos, and nothing above the cap. A sharp corner at the cap would switch the pull off there while the anti-subspace term keeps pushing the state back down, so a state near the cap could flip between pulled and not pulled from step to step. So the corner is rounded with a softplus, a smooth version of max(0, x), of width 0.05, and the pull fades out over a narrow band around the cap, which gives the two terms a smooth point to balance at. We expect the pulled states to settle near 0.8, keeping a part off the axis for the projection to land on. Measurements use the raw cosine.

What we expect. We do not predict either way. The pooled term concentrates the pull where alignment comes cheapest, which argues for saturation; the anti-subspace term penalizes the squared alignment of every state, which argues against it, and red under the same recipe reached about 0.5 at its own token. What we do expect is that the query ? carries more alignment than any example position on the whole-line arm, since it is the first position at which the whole context has been seen and its state is free.

Not reached. Rule (a) did not pass, so this is not scored. The arms trained, and their eval is in the store.

The verification rule (d)¶

The rule. Verification lines stay in the corpus from round 3 on if they leave completion unchanged: the seed-mean held-out expected exact match on completion contexts is within the seed band of the arm without them, on both pairs, control-verify against control-k3-r0.3 and anchor-verify against anchor-whole. Both pairs have to hold; a corpus that costs the anchored arm and not the control would say the anchor and the second task compete.

A verification line replaces the completion of 30% of contexts with a candidate answer, a marker, and a TRUE or FALSE verdict (the pivot); the loss is masked on a FALSE candidate (the answer token only). The verification accuracy itself (the share of verdicts right, held out) is reported with no gate, as the first number D2.3 gets; so is the op margin of anchor-verify against the margin of the whole-line arm.

What we expect. Completion is unchanged on both pairs. The verification arms see 70% of the completion lines the others do, over the same steps, and the calibration look in ex-2.2.9 found that the task is limited by distinct lines rather than passes over them; so the cost, if there is one, shows on the ops that round the most. A miss on the control pair would send the rate down before round 3 rather than take verification out.

Not reached. Rule (a) did not pass, so this is not scored. The arms trained, and their eval is in the store.

The operator rule (e)¶

The rule. A scoring-only suppression pass runs on the checkpoints of the pilot itself, on the whole-line arm, the hinge arm, and the control at the center: the projection along its dose axis (γ in 0.25, 0.5, 0.75, 1: the fraction of the component on e₁ removed), the repulsion along its dose axis (a landing at 0.25, 0, for states above an alignment of 0.5), and the reflection (the projection at γ = 2, one dose, as a reference). Each acts at every slice, at three sites: the query ?, the query =, and every position. The rule is scored on the every-position edit, on the anchored arm that rule (c) sends forward: the whole-line arm, or the hinge arm if the whole-line arm saturates. The other anchored arm is reported beside it. An operator grades with dose when the seed-mean drop in expected exact match on the held-out difference contexts is non-decreasing along its dose axis and reaches, at full dose, at least 50% of the way from the clean score to the target null; it is selective when on each of the other ten ops the seed-mean drop is at most 0.02 at every dose, with the control under the same edit as the reference. The operator that does both goes forward with its dose axis; if both do, the projection, which is the simpler; if neither does, the operator for round 3 stays open, and the pilot reports why.

The target null is the answer a model that has lost difference and nothing else would give: the posterior-weighted answer distribution with difference removed and the rest renormalized. Its expected exact match on a context is the target the full-dose damage is measured against, per context, from the posterior of the method section.

What we expect. The projection grades if the query ? does not saturate, and collapses (all of its damage in the last step of γ) if it does; the repulsion grades either way, since its dose is where the state lands rather than how much is removed. Selectivity should hold for both at the query sites, which the other ops' contexts share only as syntax, and is the open question at every position, where the edit touches states that carry the other ops. The two query sites are the first bypass measurement: an edit at ? that does less than the same edit at = says the model reads the op from the examples rather than from ?.

Not reached. Rule (a) did not pass, so this is not scored. The arms trained, and their eval is in the store.

The anchor follows the evidence¶

What we expect. On the whole-line arm, the alignment at the query = on a held-out difference context rises with the posterior on difference across the middle band (0.5 to 0.95): binned by posterior, the seed-mean alignment at the last block is higher in each bin than the one below it. That is the graded stimulus, the part redness played for red, and it is a result the pull did not train for: the label is binary on the true op, so a context whose examples half-fit difference was pulled as hard as one that names it. A shortcut, such as a characteristic answer color, would likely not follow the posterior.

Variant (c) leaves the middle-band contexts unlabelled, so grading there under (c) is the cleaner version of the same result; variant (d) trains it and is the comparison. Alignment against the posterior on the other ops' contexts (where the posterior on difference is low but not zero) is reported beside it, since a graded response should be low and flat there.

Not reached. Rule (a) did not pass, so this is not scored. The arms trained, and their eval is in the store.

The leans under whole¶

What we expect. On the whole-line arm, both leans stay within the seed band of the control. Ex-2.2.15 found that under all, lines cut short by the window taught the model to lean toward e₁ at the first operand of every line, and more so on a trailing fragment shown alone; under whole both leans sat inside the band of the control. This grammar gives a cut context more to lean with, since a window cuts more contexts than lines and a fragment of a context still looks like a context, so the lean is worth reading again. The first-operand lean is ᾱ at the first token of the context at the last block; the trailing-fragment lean is ᾱ over every position, at the last block, of a context cut to start at each example boundary, fed on its own; both over the held-out contexts of every op, with the mask arms measured with the mask on.

Not reached. Rule (a) did not pass, so this is not scored. The arms trained, and their eval is in the store.

Discussion¶

The pilot expected the d64-L4 control to reach its ceiling at every condition, on the reasoning that inferring the op is a lookup over eleven ops per example and a pool over examples. It got a little under halfway instead, at every condition and at twice the width. Validation loss was still falling at the end of training, so the first question after the pilot was whether the recipe or the task was the limit.

Ex-2.2.17 is a scout on that question, on the center control. A newline mask, a lower peak learning rate, and eight times the steps lift it to about 0.45, still short of the pass line, and no schedule, length, width, or depth moves it further. Its scoring places the remaining gap on contexts whose examples settle the op, and on one op, hsvmix, that the model computes poorly. Its backlog items on the op set (similar ops, hsvmix) are the grammar side of the rework.

None of rules (b) to (e) was tested here, so they can carry over to the next pilot unchanged.

Method¶

The posterior over ops¶

Each example is a pair of operands and an answer, and under stochastic rounding the answer is a draw from up to eight colors. So the likelihood of a shown answer y under an op o is the probability that rounding the result of o on that pair gives y, written \(P_o(y \mid a, b)\) (answer_dist in sca.data.ops).

The corpus rounds each channel independently, in proportion to where the raw value sits between grid levels. The posterior uses the same rounding.

Replacement op noise enters the likelihood as a mixture. With rate ρ, each example shows a draw from the distribution of another op instead, the other op uniform over the ten, so

where \(L_o\) is the likelihood of the shown answer under op o in the noisy corpus, \(\mathbb{E}_{o' \neq o}\) is the mean over the ten other ops, and κ is the cube-noise rate. The posterior over the eleven ops is the product of the example likelihoods under a uniform prior. This is the posterior a model trained on the noisy corpus can reach at best. It also keeps every op above zero whenever ρ or κ is above zero, which is what lets the target null weight the other ops by how nearly they fit on a context that fits one op alone.

Of two possible readings of ρ, we take the one whose numbers match the pivot: per example, with the replacing op uniform over the other ten. A replacement is invisible when the replacing op agrees with the true op on the pair, so slightly fewer than a fraction ρ of examples mislead, and the posterior is a smooth weighting rather than a count of the ops that fit.

Under this reading, the shares of contexts in each band of the posterior reproduce the scratch simulation in the pivot: one clean example puts the posterior above 0.95 on 47% of contexts and three do so on 91%; at three examples and ρ = 0.35, 21% are above the band, 45% inside it, and 34% below.

The figure covers the whole grid, 6 example counts by 7 replacement rates, with the true op uniform over the table.

Clean examples pin the op down fast, so the example count alone grades very little; replacement op noise is what spreads the posterior across the range. The spread is widest, at a standard deviation of 0.35, for 3 examples at ρ = 0.4; at the center condition it is 0.33, with 45% of contexts in the middle band and 27% below it.

The Bayes ceiling and the floor¶

A model that has learned everything the examples say answers the query with the answer distribution weighted by the posterior, \(q(y) = \sum_o \pi_o P_o(y \mid c, d)\) for the query pair \((c, d)\). That is the distribution cross-entropy training converges to, and a calibrated model holds it.¹

Our reports score expected exact match: the probability mass the model puts on the answers the true op can give, weighted by how often it gives them. So the ceiling is \(\sum_y q(y)\, P_t(y \mid c, d)\) under the true op t, averaged over contexts.

Told the op, the same predictor scores \(\sum_y P_t(y)^2\), which is 0.702 over the table. The shortfall from 1 comes from stochastic rounding, and it does not depend on the examples or the noise.

The floor (no evidence, uniform over ops) is the same at every grid point, 0.109 (the sampled values run from 0.108 to 0.109).

With three clean examples the ceiling is 0.692, within 0.02 of a model told the op, so almost all of the shortfall is rounding. Replacement op noise adds a shortfall that comes from inference: at three examples and ρ = 0.35 the ceiling falls to 0.485, and at the center condition it is 0.530.

The numbers in the pivot are the hard-accuracy form. The pivot quotes a ceiling of about 0.74 at three clean examples, 0.58 at ρ = 0.35, and 0.75 for a model told the op. Those are what the same predictor scores when it puts all its mass on the mode of \(q\): 0.741, 0.586, and 0.745 here.

That form is higher because expected exact match is linear in the model distribution, so the metric itself is maximized by a model that names one color. The calibrated predictor spreads its mass over every rounding of the answer, and scores about 0.05 less at three clean examples and 0.1 less at ρ = 0.35.

The ceiling of record is the calibrated one, since that is where training aims. A control scoring above it would have sharpened past calibration, and the calibration check in the measurements would show that. The tables keep the hard form as the maximum of the metric.

Per op, the ceiling is capped by how much the op rounds. difference is total on the grid, so its told-op ceiling is 1 and its ceiling at the center condition is 0.776, with the posterior on it spread about as widely as on any op (sd 0.32, 43% in the middle band). That is the graded stimulus the anchored arms are scored against.

Cube noise¶

The pivot allows cube noise at a low rate, so that the model learns to discount examples that fit nothing. At the center condition it costs 0.013 of ceiling at κ = 0.02, 0.031 at κ = 0.05, and 0.063 at κ = 0.1.

A posterior that does not know about cube noise treats a cube color as replacement op noise. It is a little sharper, and the metric rewards sharpness (as above), so it scores 0.006 higher at the highest rate. So the ceiling barely depends on whether the model has learned about cube noise. Whether the corpus carries it is left open (CUBE_RATE).

The three corpus conditions¶

Rule (a) picks the widest spread among the conditions where the control nears its ceiling, so the three should differ in spread and in ceiling, and bracket the working point of the pivot of three examples at ρ near 0.3. We propose k3-r0.2, k3-r0.3, and k4-r0.3.

k3-r0.3 is the center. k3-r0.2 steps ρ down on the same line length: it gives up spread (0.28 against 0.33) for a higher ceiling, and is the fallback within the same block size if the control falls short at the center.

k4-r0.3 adds one example at the center ρ, which raises the ceiling by 0.06 and keeps most of the spread (0.29), though fewer contexts sit in the middle band (31% against 45%). It asks whether more evidence buys a higher ceiling without flattening the stimulus, at 30 tokens per line rather than 24.

Two alternatives were weighed. k3-r0.4 has the widest spread on the three-example row, but its middle-band share (43%) is no larger than at the center and its ceiling is 0.09 lower, so it grades no better and leaves less room between floor and ceiling. k4-r0.35 comes closer to the spread at the center (0.32, with 35% in the middle band) at a ceiling of 0.548; k4-r0.3 gives up a little of that spread for 0.04 more ceiling.

Reading a score against its ceiling¶

The ceiling differs between conditions, so a score is shown two ways. Figures keep raw expected exact match on the y-axis and draw the ceiling and floor of each condition as dashed lines beside its seed marks, so the reader sees without arithmetic how much of the available room a model takes.

A table that compares across conditions adds the skill score as a column (0 is a model that ignores the examples, 1 is the Bayes predictor). The figure for the control also draws the ceiling for a model told the op (0.702) beside the Bayes ceiling, so that the part of the gap due to inference can be seen.

The calibration check¶

The check compares the model's answer distribution at the query =, \(p\), with the Bayes predictive \(q\) on the same context, through the KL divergence KL(q ‖ p) = Σ_y q(y) log(q(y) / p(y)), averaged over held-out contexts. Averaged that way it is the cross-entropy of the model on the answer less the cross-entropy of the Bayes predictor, since the true answers are drawn from \(q\) once the true op is marginalized under the posterior. So the number is excess loss, in nats, and zero means the model holds exactly the distribution the examples support.

The table below gives the scale, at the cube-noise rate of record. H(q) is the irreducible loss: what even the Bayes predictor pays, from rounding and from the ops the examples do not rule out. The other columns are the excess of four predictors that miss calibration in named ways: the floor predictor (no evidence), one that commits to the most probable op and answers with its distribution, one that ignores one of the examples, and one that mixes a tenth of the floor into the Bayes answer.

Two things to take from it. Ignoring one example costs several times the threshold of 0.05, so a calibrated model is one that uses all of its examples; and committing to one op costs more than ignoring an example, so a model that names the op instead of weighing them shows up here even where its expected exact match is high (that is the sharpening the ceiling of record allows for). The mix column is what the threshold roughly means: less miscalibrated than a Bayes predictor with a tenth of ignorance stirred in.

The ceiling of record at each condition is the Bayes ceiling at κ = 0.02, from the cube table. The floor does not depend on the examples, so it is the κ = 0 value.

The corpus and the training¶

One context per line, the op uniform over table A+, every pair uniform over the ordered pairs of the grid, and the query clean; the noise model is the one posterior.py assumes, a per-example replacement rate ρ with the replacing op uniform over the other ten and cube noise uniform over the grid at κ = 0.02, so the ceiling here is the ceiling of the corpus. The generator is sca.data.incontext; a context of k examples is 6k + 6 tokens, so a line is 24 tokens at three examples and 30 at four, and 26 at three with a verification line.

The corpus holds 300,000 contexts, the line count of ex-2.2.14, so difference has about the same number of lines and the labeller the same number of draws (about 545 labelled contexts). The training window is 96 tokens, a random crop of the packed corpus, which holds about two whole contexts at four examples and three at three, with a fragment at each end. Held out: 2,000 contexts per op, drawn from the same generator at a seed the corpus does not use.

The line and role of each token come from the positions of ⏎ (line_role_arrays), since the length of a context depends on its example count and on whether it carries a verification line. The labeller draws once per context from the op array beside the corpus. Crop policy whole pulls a labelled context only when the window holds all of it.

The recipe is that of ex-2.2.14, unchanged: λ_a = 0.1 annealed to a 0.1 floor over the last tenth of training, τ = 0.1, the anti-subspace weight from 2.5× the anchor weight to 0.3× by 90% of training, the untied readout, 50 epochs at d64-L4. A context is four times the tokens of an op-word line, so an epoch is about four times the steps of an epoch in ex-2.2.14.

The measurements¶

Every arm is scored at the end of training on its held-out contexts, one context per forward pass, with the mask arms measured with the mask on.

• The task: held-out expected exact match, over all contexts and per op, against the ceiling and floor of record for the condition of the arm. Rule (a) scores the controls on it; rules (b) and (d) score the center arms on it.
• The calibration check: the mean KL divergence from the Bayes predictive to the model's answer distribution at the query =, as above, on every arm; every figure that plots expected exact match plots it beside.
• Alignment by role and slice: the mean cosine with e₁ at every role of the context (the operands, ?, =, and the answer of each example, and of the query) at each of the five slices, on the difference contexts and on the other ops', with the query ? and the query = read separately. Rule (c) uses the query ? at the last block.
• Alignment against the posterior: the alignment at the query = and the query ? at the last block, per held-out difference context, beside the posterior on difference for that context, from the method section. The evidence section bins it.
• The op margin: the statistic of ex-2.2.14 with the contexts of the anchored op as the labelled group and roles running over the whole context, on the same held-out contexts. Rules (b) and (c) take it as a share of the margin of the whole-line arm.
• The leans: the first-operand lean (ᾱ at the first token, last block) and the trailing-fragment lean (ᾱ over every position, last block, of a context cut to start at each example boundary, fed on its own), over the held-out contexts of every op, as ex-2.2.15 measured them. The op margin and both leans are also recorded every 50 training steps on a fixed probe set, as ex-2.2.15 did, so the report can say how each arm got where it landed.
• The suppression pass: on the whole-line arm, the hinge arm, and the control, each operator at each dose and site, the held-out expected exact match on difference contexts and on those of each other op, against the clean pass and the target null per context, and the KL divergence from the target-null predictive to the edited answer distribution. Rule (e) is scored on it.
• Verification accuracy on the verification arms, held out, with no gate.

Figures label the roles in the notation of the posterior section: a, b, and y for the operands and the answer, subscripted by example, with ? and = as themselves.

Budget¶

42 runs: 14 arms at 3 seeds. Ex-2.2.14 trained 50 runs of 50 epochs at d64-L4 on a 300k-line corpus for about $1.80 on Modal, and ex-2.2.15 trained 55 runs for about $2.30 (bin/mini cost): about 3.5 cents and two to three minutes on an L4 per run. At this model size a training step is bound by latency, not arithmetic: about 10.5 ms on an L4 whatever the window holds (GPU notes), and an ex-2.2.14 run took about 5,000 steps. So the cost follows the step count. A context here has four to five times the tokens of an op-word line and the window is half as long again, so an epoch is about three times the steps: some 13,000 steps at three examples and 16,500 at four, which is two and a half to three minutes of stepping, or four to six minutes per training task with compilation, start-up, and the trajectory reads, well inside the one-hour timeout. At the per-step cost of ex-2.2.14 that puts the training at $3 to $4, a little more for the d128-L4 control. The window of 96 tokens at d64-L4 fits an L4 with room to spare. Eval adds the alignment measurements on eleven held-out sets, the posterior on every held-out context (a table lookup), and the suppression pass, which is scoring only: 3 sites × (4 + 2 + 1) edits on nine checkpoints. Under $6 in all, at the pace --max-containers 12 --budget 6h sets. The training trajectories record the margin and the leans every 50 steps, so a corpus that saturates early shows there, and a later round can train on fewer contexts.