Ex 2.2.17: the center control plateau, a scout¶

This scout set out to lift ex-2.2.16's center control (the unanchored d64-L4 model on the corpus condition k3-r0.3) to its Bayes ceiling of 0.516. Ex-2.2.16 trained it for 50 epochs at a peak learning rate of 0.01, and it reached 0.27. More steps, a newline mask, and a lower rate took it to about 0.45. Past that, neither the shape of the schedule, nor twice the steps, nor a wider or deeper model changed the final score by more than the spread between seeds. The best run, sweep-0.00316-s1, reached 0.464.

To see where the rest of the gap sits, we scored the answer distribution of each model against the Bayes predictive, context by context. Part of the gap is in how firmly the model commits to the op its examples support, and hsvmix adds a separate shortfall.

Observations¶

Each line is a measurement on the runs of this scout, with no gate.

• E1 More steps, a lower rate, and the mask: more steps and the newline mask made most of the gain, and a range of peak rates did about equally well.
• E2 Schedule, length, and model size: past that, neither the shape of the schedule, nor more steps, nor a wider or deeper model moved the final score.
• E3 Where the gap sits: the model matches the ceiling on contexts whose examples leave the op uncertain, and falls short on those that point to one op.
• E4 The leak onto other ops: where the examples settle the op, the model keeps some of its mass on answers of other ops, mostly the op most like the true one, and more than nearness to the true answer accounts for.
• E5 Answers in the color cube: for ten of the eleven ops, the expected answer of the model sits close to the Bayes one, with no shared direction to the difference. hsvmix is the exception.
• E6 hsvmix and the hue gap: hsvmix falls further short of its ceiling the further apart the operand hues are, and neither width nor depth changed that.

Scope¶

This is a scout, run in rounds, each designed on the results of the one before. There is no preregistration and no gate. The pass line that ex-2.2.16 set for its controls, 0.03 below the ceiling at 0.486, is a reference point here. Most conditions after round 1 have one seed, and the three-seed conditions span about 0.03, so a difference smaller than that is not a change.

Why¶

The anchoring experiments on the in-context grammar compare an anchored model with an unanchored control on the same contexts. At 0.27 against a ceiling of 0.52, ex-2.2.16's center control had learned about half of what its examples allow, so that comparison would mix the effect of the anchor with the unfinished training of both models. We wanted a control recipe that comes close to the ceiling, or else to know where it levels off and why.

The runs¶

Every run is the unanchored model of ex-2.2.16 on k3-r0.3, with an untied readout. Lengths are multiples of the 50 epochs (13,200 steps) that ex-2.2.16 trained for.

The newline mask stops each position from attending past the start of its own line. Every schedule starts with a linear warm-up from 1% of the peak rate: over the first tenth of the run in round 1, and over 1,320 steps (5 epochs, ex-2.2.16's own warm-up) from round 2 on, whatever the length. The cosine then anneals to 1% of the peak by the end of the run.

The measurements¶

Held-out expected exact match (EEM) is the probability that an answer drawn from the distribution of the model at the query = is a correct answer of the true op. We average it over 22,000 held-out contexts, 2,000 per op.

The Bayes ceiling is the same score for an ideal predictor. That predictor weighs each op by how well it explains the examples of a context, then answers with the resulting mixture of ops; we call that mixture the Bayes predictive. The ceiling is below 1 because some ops round stochastically, and noisy examples leave the op uncertain.

The calibration KL is the KL divergence¹ from the Bayes predictive to the distribution of the model, in nats. It measures how much more the model loses on the answer than the ideal predictor does. Zero means the model holds the distribution the examples support.

The posterior on the true op is the probability the ideal predictor gives the true op after seeing the examples. Replacement op noise lowers it. We call a context confident when that posterior is above 0.99.

The support of the Bayes predictive on a context is the set of grid colors to which it gives at least 0.02. Probability mass on any other color is off support. An off-support color is another op's answer when some op, applied to the query operands, can give it. Distances are on the color grid, in grid steps: one step is the spacing between neighboring levels of a channel, and the far corner of the cube is 5√3 ≈ 8.7 steps from black.

More steps, a lower rate, and the mask (E1)¶

Round 1 tested two ideas at three seeds each, without the newline mask: that the peak rate of 0.01 was too high, or that the model needed more steps. The low arm trained for the length of ex-2.2.16 at 0.003, long for three times the length at 0.01, and long-low did both. Neither change alone did much, and the two together did the most.

Every round-1 curve was still rising when the cosine schedule wound down. So round 2 trained for eight times the length (105,600 steps), with the mask. We ran a learning-rate finder on the same configuration. It trains briefly at a rising rate and looks for where the loss falls fastest; here, the steepest descent was near 0.0031. Round 2 then swept the peak rate from 0.01 down to 0.00032 at one seed each, with one more run at 0.00316 without the mask. The figure below follows each run through training, one panel per group of ops, with the learning rate it followed underneath.

Four peak rates from 0.00178 to 0.01 ended within 0.04 of each other, and the best was 0.00562 at 0.446. The mask made the largest single difference in the scout: at 0.00316, the unmasked run scored 0.334 and the masked one 0.436. The HSV-channel ops rose steeply once the decaying rate fell below about 0.004, which shaped the next two rounds. From round 3 on, every run trains with the mask at a peak rate of 0.00316, the middle of the good range.

Schedule, length, and model size (E2)¶

Round 3 tried a warmup-stable-decay (WSD) schedule, which warms up to 0.00316, holds near 0.0025 until 85% of the run, and then anneals to the same floor as the cosine. Round 4 tried a staircase: holds at 0.0021, 0.001, 0.0001, 1e-05, with short anneals between them, the last hold covering the final 11% of the run. Both ran at three seeds, beside two more seeds of the cosine.

The schedules change the path and leave the destination in place. WSD curves stay flat through the hold and rise steeply in the final anneal; the staircase gains most at its first two drops, and almost nothing at the lowest rate. All three end together, and the seeds differ more than the schedules do.

Round 5 trained each schedule for sixteen times the length (211,200 steps) at one seed. Rounds 6 and 7 trained the cosine at eight times on a wider model (d128-L4) and a deeper one (d64-L6), one seed each, both at seed 0 so that they pair with the other seed-0 runs. The figure below gathers the final scores of all of these.

At eight times the length, the nine runs of the three schedules span 0.436 to 0.464. Sixteen times the length adds between 0.002 and 0.006 to the seed-0 run of each schedule. The wider and deeper models gain 0.023 and 0.017 on their seed-0 pair, but seed 0 is the weakest seed under every schedule, and the plain model at seed 1 reaches 0.464.

Two things about the wider model stand apart from its final score. It learns faster: its HSV channels make their jump near 15k steps, against 25k to 45k for the d64 runs, and it leads through the first half of training. And it is much less well calibrated, at a KL of 0.95 against about 0.5, with a lower training loss than the d64 runs. So it is about as accurate as the others while being more confident than the examples support, which fits a model that has fitted more of its training corpus.

Where the gap sits (E3)¶

To see where the rest of the gap sits, we scored the final answer distribution of every masked run at 0.00316, context by context (see Method). The figure below groups the held-out contexts two ways: by the posterior on the true op, and by the number of examples that carry replacement op noise.

Where the examples leave the op uncertain (a posterior below 0.5, or two or more noisy examples out of three), the model scores what the ideal predictor scores. The gap opens once the examples point to one op: 0.098 in the band from 0.5 to 0.9 and 0.097 from 0.9 to 0.99, and it narrows to 0.056 where the posterior is above 0.99. Those two middle bands hold 53% of the contexts and 79% of the gap. All 14 runs share this shape, whatever their schedule, length, or model.

The leak onto other ops (E4)¶

A model that knew the true op would lose nothing to op uncertainty. So on confident contexts, what remains of the gap is about how the model computes each op, or about the model spreading its mass over ops the examples have ruled out. Where the model puts its off-support mass tells the two apart. The table below splits that mass by whether another op would give the color on the same query.

On every op, the model keeps between 6% and 28% of its mass off the support, and more than half of that sits on answers another op would give, where an even spread would put under a tenth.

lighten and darken are the clearest case. Each gives one answer per query, with no rounding, and on a confident context the examples have ruled out the other ops, so the Bayes predictive puts all its mass on that answer and the ceiling is 1. Over all their contexts, where noisy examples often leave the op uncertain, their ceilings are 0.71 and 0.69. On their confident contexts the top answer of the model is the Bayes answer 98% of the time or more, and still about 10% of its mass goes to other ops' answers.

Other ops' answers are often near the true answer in the cube, and some ops give similar answers (lighten and screen both brighten), so part of this mass could be near misses. To separate the two, we compare colors at the same distance from the support. If nearness alone set the mass, a color that is another op's answer would get the same mass as one that no op gives at that distance. We also repeat the comparison inside the box the two operands span, since several ops give answers between their operands.

At every distance, the colors that are other ops' answers get more mass: 3.9 times as much one step from the support, and more further out. Nearness does shape the leak, since 60% of the mass on other ops' answers sits one step from the support. The operand box also matters: at the same distance, a color no op gives gets 2.5 times as much mass when it lies inside the box as off-support colors do on average. But inside the box, other ops' answers still get 3 to 15 times the mass of other colors at the same distance.

So the model favors the answers of the ops its examples should have ruled out, beyond what nearness or the operands account for. The runs that leak more also score lower: across the 14 runs, the mean leak on other ops' answers correlates with the final score at r = -0.94.

To see which ops the leak goes to, the figure below breaks it down by op. For each true op, it counts the mass on answers that another op can give and the true op cannot.

Most of the leak goes to one other op, the one most like the true op. On confident contexts, lighten puts 0.06 of its mass on answers only screen gives, and darken puts 0.05 on answers only multiply gives. hsvmix and mix leak onto each other (0.06 and 0.05), and difference onto exclusion (0.03). For every op, the op it leaks onto most is the one whose answers most often coincide with its own: the answer of a lighten query is one that screen can give on 90% of confident contexts, and for darken and multiply the figure is 88%. The leak runs mostly one way, from lighten to screen and from darken to multiply (0.01 and 0.01 in the other direction).

So where two ops agree on most queries, the model keeps part of its mass on the answers of the other op even after the examples have told them apart.

Answers in the color cube (E5)¶

The cubes below put the answers of the model beside the Bayes answers, one panel per op, on the confident contexts of sweep-0.00316-s1. Each context is drawn at its expected answer color: the mean color of the answer distribution, which moves toward the middle of the cube as mass spreads.

None of the stubs is long. For ten of the eleven ops, the expected answer of the model sits within a fraction of a grid step of the Bayes one, with no shared direction to the difference. So the model has learned these ops well, and the leak above is spread thinly around the cube. hsvmix is the exception, as the ops table shows: it keeps 28% of its mass off support on confident contexts, and 12% goes to colors no op gives. It is the one op where the computation itself falls short.

hsvmix and the hue gap (E6)¶

hsvmix mixes two colors in hue, saturation, and value, with the hue taken around the color wheel. Grouping its contexts (posterior above 0.9, two chromatic operands) by how far apart the operand hues are shows where it struggles.

With nearby hues the model gets 83% of what the ceiling allows; with near-opposite hues it gets 43%. Near-opposite hues are where the midpoint around the wheel moves furthest for a small change in either operand, and at exactly opposite hues the direction around the wheel is a tie. So hsvmix asks for a more precise computation than the other ops, and the model has not learned it. Width and depth did not help: hsvmix scored 0.20 on d128-L4 and 0.19 on d64-L6, against 0.17 on their seed-0 pair.

What we make of it¶

The control now sits at about 0.45 on the center condition, with the mask, a peak rate of 0.00316, and eight times the steps of ex-2.2.16. That falls short of the pass line, but it is a model that does in-context inference over ops. It tracks the ceiling across the whole range of evidence, its answers land close to the Bayes answers on ten ops, and its shortfall is concentrated in how firmly it commits once the op is clear, plus one op it computes poorly. That seems a workable baseline for the anchoring experiments, which compare an anchored model with this control on the same contexts.

Four questions from this scout are on the backlog. Two are about the recipe: whether a curriculum over the replacement rate changes how firmly the model commits (noise curriculum), and whether the wider model with a tuned rate can reach the same level in half the steps (a cheaper recipe); the wider model learned faster here, which is some encouragement for the second. Two are about the op set: whether to drop one op of each pair whose answers often coincide, since that is where most of the leak goes (similar ops), and whether hsvmix needs more training on hue, saturation, and value, or should leave the op set (hsvmix).

Method¶

Scoring. For every held-out context we score the full answer distribution of the model at the query =, over the 216 color tokens, for the 14 masked runs at 0.00316: every run of rounds 3 to 7 and the round-2 cosine. We then redraw the held-out contexts from their seed to recover which examples carry replacement op noise, which ex-2.2.16 did not publish; the redrawn contexts match the published ones token for token. The answer table, the posterior, and the Bayes predictive come from ex-2.2.16.

Schedules. WSD and the staircase are dopesheets: tables of the rate, as a multiple of the peak, at points through the run, stretched to its length. Each warms up over its first 1.25% (1,320 steps at eight times, twice that at sixteen). The learning-rate panels are drawn from the schedule of each run, which matches the rate logged during training.

Distance to the support. For each confident context and each off-support color, the grid distance to the nearest color in the support, binned at one step, √2, √3, 2 to √5, √6 to 3, and beyond. Within a bin, the mass per color is the mean over every pair of context and color in it. The operand box holds the grid colors that lie between the two operands on every channel.

Op confusion. For each confident context, the grid colors each op can give for the query operands. The diagonal counts the mass on the colors of the true op, and an off-diagonal square the mass on colors of the column op that the true op cannot give; the overlap quoted in the text counts every color of the column op, on the Bayes predictive.

Cube figures. Up to 90 contexts per op (or per hue-gap group), a fixed sample from sweep-0.00316-s1; any run would do, since the runs agree closely on every summary above.

Budget. The scout cost about $15 on Modal, nearly all of it L4 time for training. A step takes the same time on d64-L4 and d128-L4 (about 2,000 to 3,000 steps a minute), since at these sizes the step is bound by latency rather than arithmetic, so the cost follows the step count. An eight-times run is about 40 minutes of training.