Ex 2.2.19: training length and seeds for the seven-op set¶

Findings¶

• The scout (S1) — the rule picked 200 epochs: at the scout seed that run fell short of the 400-epoch run by 0.008, against 0.015, with no op over its tolerance. The 50- and 100-epoch runs fell short by 0.167 and 0.124.
• A shorter run keeps most of the skill (H1) — partial pass. On the fresh seeds the 200-epoch run falls short by 0.0157 on average, just over the gate of 0.015 and inside the partial band to 0.03. No op is over its tolerance, though hue-hsv is within 0.0002 of it. We adopt 200 epochs.
• The seven-op set stays closer to its ceiling (H2) — pass. no-four has the smaller gap at all three paired seeds, 0.037 to 0.046 against 0.053 to 0.074 for full.
• Skill curves (E1) — the HSV-channel ops take off at about the same epoch in the 200- and 400-epoch runs (80 to 112). So they come late in the shorter run, and not at all in runs that end before that epoch.
• Calibration (E2) — calibration KL at 200 epochs (0.314 to 0.336) matches 400 epochs (0.317 to 0.351).
• Op confusion (E3) — the 200-epoch run puts more mass off the diagonal in every row, mostly in the HSV-channel rows, with no single similar op taking it.
• Spread over seeds (E4) — the EEM range over four seeds is 0.013 at 200 epochs and 0.008 at 400, against 0.028 over the three full seeds.

Why¶

Ex-2.2.18 trained the center control on several smaller op sets, one run each. Dropping screen, multiply, hsvmix, and exclusion gave a seven-op set, no-four, with a higher Bayes ceiling (defined below). The model came within 0.044 of that ceiling, against 0.074 for the full set. But that was one seed. The three ex-2.2.17 seeds of the same recipe on the full set had gaps spread over 0.028, so the difference could be seed variation.

We will probably adopt no-four for the anchoring experiments that follow, and each of those costs more the longer its runs: about $0.26 for 400 epochs.

The skill curves of ex-2.2.18 leveled off well before the end: the last fifth of training added at most about 0.015 of held-out expected exact match, which suggests a shorter run could do nearly as well. But the learning-rate schedule is a cosine, so a shorter run anneals sooner,¹ and its curve is not just a truncated longer one.

A scout trains no-four at shorter lengths from the initialization of the ex-2.2.18 run, and a rule fixed in advance picks a length. Three fresh seeds then train at that length and at 400 epochs, and the preregistered comparison uses those seeds alone: the scout seed helped pick the length, so it would flatter the pick.

The fresh seeds also give the first replicates of no-four at 400 epochs, so we can check the narrower gap of ex-2.2.18 as well.

The runs¶

Every run uses the recipe of ex-2.2.17 and ex-2.2.18, on the no-four corpus that ex-2.2.18 built, with a different number of epochs. At each shorter length the scout also tries a higher peak learning rate, since a shorter run may want one. The warmup (the opening stretch in which the learning rate ramps up to its peak) stays at 5 epochs at every length.

Model seeds 601 and 602 are the initializations of ex-2.2.17's sweep-0.00316-s1 and sweep-0.00316-s2, and seed 600 of sweep-0.00316-s0 and of the ex-2.2.18 full run. So each 400-epoch no-four run at those seeds has a full run that started from the same weights. Seed 603 has no such partner.

Held-out expected exact match (EEM) is the probability that an answer drawn from the model at the query = is a correct answer of the true op. The Bayes ceiling is the same score for an ideal predictor, one that weighs each op by how well it explains the examples. The gap is the ceiling minus EEM. All three are computed on the op set of the run, as in ex-2.2.18.

Given a fixed corpus, the ceiling is the same at every length, so a difference in gap between two lengths is a difference in EEM. A run falls short of another by how much lower its EEM is. We compare runs that share a model seed, and average those paired differences over seeds.

The scout (S1)¶ ~ Partial

The scout is a procedure, with no hypothesis; the rule below picks the length that stage 2 confirms.

The rule. At each scout length, take the run at the peak rate with the higher EEM. Of those runs, pick the shortest, T, that falls short of the ex-2.2.18 run by at most 0.015, with no op falling short by more than its own tolerance (below). If no scout length passes, stage 2 trains only the 400-epoch runs, and H1 is unresolved.

Stage 2 confirms T, and also 2T when 2T is shorter than 400 epochs, so that a lucky pass at T still leaves a length to adopt. Each trains at the rate taken at its length.

The overall tolerance is the largest last-fifth gain of any ex-2.2.18 run. The tolerance for each op is its seed range in the three ex-2.2.17 runs, or 0.03, whichever is larger, since some ops varied little across three seeds and a single scout run is noisier than that.

What we expect. At 50 epochs the run falls short by more than the tolerance: ex-2.2.17 needed several times that length for the HSV-channel ops. We don't have a strong expectation between 100 and 200 epochs. We expect the higher rate to help more the shorter the run, and at 200 epochs to make little difference. The HSV-channel ops may be the exception: in ex-2.2.17 they were learned while the rate passed down through about 0.0025, during the cosine, so a higher peak only delays that stretch, and a short run has the least time to spare.

What we saw. The rule picks 200 epochs. At 50 epochs the run falls short of the 400-epoch run by 0.167, with all 7 ops over their tolerance, and at 100 epochs by 0.124, with hue-hsv, sat-hsv, and value-hsv over. At 200 epochs the shortfall is 0.008 and no op is over. Stage 2 trains 200 epochs and the reference; 2T is 400, the reference length, so nothing between is trained.

The jump between 100 and 200 epochs is almost all in the HSV-channel ops. Their mean shortfall falls from 0.232 to 0.005, against 0.043 to 0.010 for the other four. This is the caveat we wrote in advance, that a short run has the least time to spare for these ops.

The higher rate helped as expected at the shortest lengths: EEM was 0.044 higher at 50 epochs and 0.017 higher at 100. At 200 epochs it was 0.022 lower, more than the little difference we expected. The HSV-channel ops were not an exception to the pattern: the higher rate lifted them by 0.036 at 100 epochs, more than the whole set, and cost them 0.034 at 200. The rate taken is therefore 0.00562 at 50 and 100 epochs and 0.00316 at 200.

A shorter run keeps most of the skill (H1)¶ ~ Partial

What we expect. On the fresh seeds, the run at T falls short of the run at 400 epochs by at most 0.015 on average, and no op falls short by more than its tolerance on average. That is a pass.

A shortfall between 0.015 and 0.03 (the seed range of ex-2.2.17) is a partial pass. A shortfall beyond 0.03, or an op beyond its tolerance, is a miss for H1, and would mean the scout result was flattered by its seed. 2T, when trained, gets a verdict by the same rule.

Which length we adopt. The verdicts inform this choice without settling it, since a rule written now can't anticipate everything that might matter: a short run that matches on EEM could be poorly calibrated (E2), or spread more widely over seeds (E4). We expect to adopt the shortest confirmed length that passes, and will say why if we choose otherwise.

What we saw. On the fresh seeds the 200-epoch run falls short of the 400-epoch run by 0.0186, 0.0182, and 0.0102 at model seeds 601, 602, and 603. The mean is 0.0157, 0.0007 over the gate and well inside the partial band. The shortfall at the scout seed, 0.0079, was smaller than any of the three, so the scout was flattered by its seed, as the design anticipated. 2T is the reference length, so there is no second length to score.

No op falls short by more than its tolerance on average. hue-hsv is closest, at 0.0298 against 0.030; the three HSV-channel ops have the largest shortfalls, averaging 0.023 against 0.010 for the other four.

The length we adopt. We adopt 200 epochs, at the peak rate 0.00316. The preregistered plan was to adopt the shortest confirmed length that passes, and to say why if we chose otherwise. H1 is a partial pass and 2T is the reference length, so no shorter length passes outright; we made this choice after seeing the results. The cost is about 0.016 of EEM, larger on the HSV-channel ops. Against that, the calibration KL at 200 epochs matches 400 epochs (E2); four seeds do not resolve a difference in spread (E4); and every later run costs about $0.13 instead of $0.26.

The seven-op set stays closer to its ceiling (H2)¶ ✓ Pass

What we expect. At 400 epochs, no-four has a smaller gap than full at each of the three model seeds where both exist (600, 601, and 602). This is a check on ex-2.2.18, with no gate, since we are likely to adopt no-four for its higher ceiling either way. If the gaps overlap, the narrower gap of ex-2.2.18 was seed variation.

The full runs are reused: those at seeds 601 and 602 are from ex-2.2.17 and trained on the ex-2.2.16 corpus, and the one at seed 600 is from ex-2.2.18, on a corpus rebuilt the same way, so the pairs share a starting point and differ in corpus draw as well as op set.

What we saw. no-four has the smaller gap than full at all three paired seeds: 0.044 against 0.074 at seed 600, 0.037 against 0.053 at seed 601, and 0.046 against 0.068 at seed 602. At seed 603, which has no full partner, the no-four gap is 0.040, within the range of the other three. The largest no-four gap, 0.046, is below the smallest full gap, 0.053.

The narrower gap of ex-2.2.18 was not seed variation. The gap is smaller at every seed, by 0.015 to 0.031, while the four no-four seeds vary by 0.008 among themselves. The pairs differ in corpus draw as well as op set, so this shows that no-four gets closer to its ceiling, and not which of the two changes is responsible.

Exploratory analyses¶

These are planned but have no prediction.

Skill curves (E1)¶

The skill curves of every run, to see whether the HSV-channel ops are learned later, or not at all, in the shorter runs.

What we saw. The 50- and 100-epoch runs end before the HSV-channel ops have been learned: their mean skill on the probe set is 0.23 and 0.35 at the end, against 0.73 and 0.81 for the other four ops. In the 200-epoch runs the HSV-channel ops are learned, but later in the schedule than at 400 epochs. A quarter of the way through, their mean skill is 0.03 to 0.09 at 200 epochs and 0.28 to 0.52 at 400, over the four seeds. They finish at 0.81 to 0.87 and 0.86 to 0.90.

The rest of this analysis is post hoc. In the 200-epoch and 400-epoch runs alike, the HSV-channel skill first passes 0.5 between epoch 80 to 112. So these ops come late as a share of a shorter run, yet at about the same number of epochs. The learning rate at that epoch differs: 0.0014 to 0.0022 in the 200-epoch runs and 0.0026 to 0.0029 in the 400-epoch runs. That points to the number of epochs, rather than the rate, as what matters. But the two lengths differ in their whole schedule, so eight runs cannot separate the two. It fits the 100-epoch run falling short on these ops, since it ends near that point.

Calibration (E2)¶

The calibration KL of each run (the KL divergence from the Bayes answer distribution to that of the model) beside its EEM, since ex-2.2.17 found a model can score well and be poorly calibrated.

What we saw. Calibration KL falls as EEM rises across the lengths, from 0.86 at 50 epochs at the recipe rate to 0.34 at 400. The higher rate brings it down sooner: 0.67 at 50 epochs and 0.56 at 100. At 200 and 400 epochs the points overlap: 0.314 to 0.336 over the four seeds at 200 epochs and 0.317 to 0.351 at 400. Within those two groups, a higher EEM does not go with a lower KL. The 200-epoch runs are no worse calibrated than the longer ones. So the case that ex-2.2.17 found, a model that scores well but is poorly calibrated, does not arise at 200 epochs.

Op confusion (E3)¶

The op confusion matrix of each run at the chosen length and at 400 epochs: the mass the model puts on the answers of each op, on confident contexts² of each true op, as in ex-2.2.17. A shorter run might keep more of its mass on a similar op.

What we saw. The shorter run puts more mass off the diagonal. Summed over a row, the mass on the answers of other ops averages 0.054 at 200 epochs and 0.035 at 400, and it is higher in every row. The rise is largest in the three HSV-channel rows, 0.073 against 0.042, and in the other four rows it goes from 0.031 to 0.040. The largest square is the same at both lengths, sat-hsv onto mix, at 0.023 and 0.013. So the extra mass sits mostly in the rows of the HSV-channel ops, and the shorter run leans hardest on the same op as the longer one. Every square stays small.

Spread over seeds (E4)¶

The spread of EEM over the four no-four seeds at 400 epochs, against the spread of the three full seeds, and the spread at each confirmed length.

What we saw. Over four model seeds the EEM range is 0.013 at 200 epochs and 0.008 at 400, and over the three full seeds of ex-2.2.17 it is 0.028. The 200-epoch runs vary about 1.6 times as much as the 400-epoch ones, which four seeds can hardly resolve, and both are below the full range. The lowest 200-epoch run, at 0.545, is below the lowest 400-epoch run, at 0.563, by 0.018.

What it means for what follows¶

The anchoring experiments that follow can train no-four for 200 epochs at the peak rate 0.00316, at half the cost. The shortfall is small, but most of it is in the three HSV-channel ops, so for an experiment that leans on them, an unanchored control of the same length is a fairer comparison than a 400-epoch number.

The HSV-channel ops set how short a run can be. In the post hoc part of E1 they took off at about the same epoch at both long lengths, which is consistent with needing a count of epochs rather than a share of the schedule. That rests on eight runs whose schedules differ throughout; lengths between 100 and 200 epochs, or a schedule that decays later, could separate the two.

Calibration does not separate 200 from 400 epochs, and four seeds cannot resolve a difference in spread, so the shortfall is the only cost we found. The scout used one seed, the confirmation three, and all runs one op set and one corpus, so we cannot say how the shortfall moves with either.

The narrower gap of no-four held at every paired seed, which supports it as the base for the anchoring runs.

Method¶

Recipe. The unanchored d64-L4 model with an untied readout and the newline mask, trained with a cosine schedule after a linear warmup, as in ex-2.2.18. The corpus condition is k3-r0.3 (three examples per context, with a replacement rate of 0.3) on the no-four op set (the eleven ops of ex-2.2.16 less screen, multiply, hsvmix, and exclusion), with the corpus, held-out set, and probe set that ex-2.2.18 built. So the runs differ from the ex-2.2.18 run only in length and model seed. This experiment rebuilds the corpus from the seed of ex-2.2.18, and its Bayes ceiling, 0.6091, equals the 0.6091 that ex-2.2.18 published, so the runs share that corpus.

Warmup and learning rate. The warmup is 5 epochs rather than a fixed share of the run, as in ex-2.2.17 from its second round, and the peak learning rate is 0.00316. A shorter run spends less of its schedule near the peak, so it may do better at a higher peak rate. The scout tries 0.00562 beside it at each shorter length: the next rate up on the ex-2.2.17 grid, which at 400 epochs scored level with 0.00316 at one seed, where 0.01 scored lower. The 400-epoch runs stay at 0.00316, so H1 compares a shorter recipe, rate included, with the recipe we have.

Measurements. EEM, ceiling, floor, and calibration KL are measured on the 2,000 held-out contexts per op, with the evaluation of ex-2.2.18. Skill curves are logged at about 50 points per run on 200 held-out contexts per op.

Per-op tolerances.