Ex 2.2.18: Dropping ops with similar answers¶

In ex-2.2.17, the center control kept part of its probability mass on the answers of a similar op, even on contexts where the examples should have made the op unambiguous. Four pairs of ops stood out: lighten with screen, darken with multiply, mix with hsvmix, and difference with exclusion. This scout drops one op from each pair, first one at a time and then all four together. A second round drops the other op of the first two pairs: lighten alone, darken alone, and both of them in place of screen and multiply in the four-op drop. Each op set trains with the ex-2.2.17 recipe once (a single seed).

Dropping ops changes the task, so every op set has its own Bayes ceiling and floor, and we score each run against the ceiling and floor of its op set.

Observations¶

Each item below is a measurement on the runs of this scout, with no gate.

• Scores against each ceiling (E1): dropping lighten or darken lowered the ceiling, and dropping any of the other four raised it. Without screen, multiply, hsvmix, and exclusion, the model came within 0.044 of its ceiling, closer than any run so far. The second four-op drop came within 0.055.
• The HSV ops (E2): three of the six single drops fell short on the three HSV ops, and the three runs without hsvmix learned those ops earlier than the others did.
• The leak onto a dropped op (E3): on contexts of a partner op, the mass the model put on the answers of the dropped op mostly went away with it.
• Training time (E4): every run reached 95% of its final skill by 82% of the way through training, and the last fifth of training added little.

Scope¶

This is a scout, with no preregistration and no gate. Each op set has one run, all from the same model seed, so a difference between two runs is only a hint. For scale, ex-2.2.17 trained this recipe on the full op set at three seeds, and their gaps to the ceiling spanned 0.053 to 0.080.

Why this experiment¶

The in-context grammar asks the model to infer the op from three examples, and some pairs of ops give the same answer on many operand pairs. An example that fits lighten often fits screen too, so those examples say less about the op, and the model has two nearly interchangeable answers to choose between. If the pairs are part of why the control falls short of its ceiling, a smaller op set might make a better grammar for the anchoring experiments.

It may matter which op of a pair is dropped, too. lighten, darken, and difference give one answer for each pair of operands, while their partners round each channel at random between grid levels, so their answers are spread over a few colors. Dropping an op with spread-out answers raises the ceiling whether or not similarity matters. Dropping its partner instead breaks up the same pair without raising the ceiling, so comparing the two tells the effect of similarity apart from the effect of rounding.

Parameters¶

Every run uses the recipe that ex-2.2.17 settled on: the unanchored d64-L4 model with an untied readout and the newline mask, a cosine schedule at a peak learning rate of 0.00316 after a warm-up of 5 epochs, for 400 epochs (105,600 steps). The corpus condition is k3-r0.3, three examples and a replacement rate of 0.3, with 300,000 contexts. Each op set gets its own corpus, holdout, and probe set, drawn from its own ops.

Measurements¶

Held-out expected exact match (EEM) is the probability that an answer drawn from the distribution of the model at the query = is a correct answer of the true op. The Bayes ceiling is the same score for an ideal predictor, which weighs each op by how well it explains the examples and answers with the resulting mixture. The floor is the score of a predictor that ignores the examples. Both are computed for each op set on its own ops, as in ex-2.2.16.

The gap is the ceiling minus EEM, and skill is how far a run got from the floor to the ceiling, as a fraction of that distance. Gaps compare runs whose ceilings differ.

Scores against each ceiling (E1)¶

The figure below puts each run beside the ceiling and floor of its op set.

The full-set run scored inside the range of the three ex-2.2.17 seeds, so the recipe reproduced. The rest of this section compares runs by their gap, since the ceiling moves from one op set to the next.

Dropping one of screen, multiply, hsvmix, or exclusion raised the ceiling, and dropping lighten or darken lowered it. That follows from which side rounds at random. Even a predictor told the op, with no inference to do, can't always name the answer of an op that rounds at random, but it always can for lighten or darken. So the ceiling rises when an op that rounds at random goes, and falls when its partner goes.

The gaps of the single drops mostly stayed near that of the full set, 0.074. Three were wider: no-multiply and no-darken, the two sides of the darkening pair, and no-exclusion. E2 shows where those fell short.

The two four-op drops break the same four pairs, and both narrowed the gap: to 0.044 without screen, multiply, hsvmix, and exclusion, and to 0.055 without lighten, darken, hsvmix, and exclusion. Only the first raised the ceiling, so most of the higher score of no-four came from its higher ceiling.

But each op set has only a single run, and the gaps of the three ex-2.2.17 seeds on the full set spread over a range about as wide as either change. So these narrower gaps could be seed variation, and it would take more seeds to tell.

The HSV ops (E2)¶

The heatmap below breaks each gap down by op.

Three single drops fell short on the three HSV ops: no-multiply, no-exclusion, and no-darken. Their mean gap on those ops was at least 0.12, against 0.06 in the full-set run. The HSV ops are blend modes that take one of hue, saturation, or value from one operand and the other two from the other. None of the dropped ops gives answers like theirs, so similarity doesn't explain this.

The learning curves suggest timing. Ex-2.2.17 saw these ops rise steeply partway through training, and in this scout the rise came at different times in different runs. At step 23,232, the mean probe EEM on the three ops was 0.36, 0.35, and 0.29 in the three runs without hsvmix, and between 0.13 and 0.23 in the others. The runs that fell short had not caught up when the schedule ended.

With one seed per op set, we can't tell whether dropping multiply, darken, or exclusion makes those ops harder to learn, or these runs were slow ones. Dropping an op also changes the corpus, since the other ops share its contexts, so each gets a little more training. The early start without hsvmix is firmer: three runs share it, though two of them also drop three other ops.

Away from the HSV ops, both four-op columns are paler than the full set on nearly every op. The gap on difference fell by about half in both, and the gaps on lighten and darken by more than half in no-four.

The leak onto a dropped op (E3)¶

Ex-2.2.17 found the model keeping mass on the answers of a similar op even where the examples should have made the op unambiguous. To see where that mass goes, we take the held-out contexts whose posterior on the true op is above 0.99, and for each other op measure the mass the model puts on colors that op can give and the true op cannot. The figure below shows this as one matrix per op set, the same measurement as the op confusion of ex-2.2.17. The Bayes predictive puts almost nothing off the diagonal on these contexts.

In the full set, most of the mass off the diagonal sits in the squares of the four pairs. Confident lighten contexts put 0.10 of their mass on colors only screen gives, and darken, mix, and difference contexts put 0.07, 0.05, and 0.03 on colors only their partner gives. The leak is lopsided: confident screen contexts put 0.019 on colors only lighten gives. Part of that is in how it is counted. lighten gives one color, often one of the colors screen gives, so there are fewer colors for screen contexts to leak onto.

A dropped op keeps its column. The model answers with colors and never names an op, so the colors screen would give are still colors the model can put mass on, whether or not it trained on screen. What changes is why mass lands there. In a run that drops an op, nothing it learned points at those colors, so its column is a baseline: the mass that falls on some set of plausible colors through the general spread of the answers. It is small. On the partner row, the leak onto a dropped op is at most 0.020, about the level of the unrelated ops on the same row, whose median square runs from 0.003 to 0.018 across these runs.

That gives a scale for the rest of the matrix. A square means something only where it stands clear of this background, so lighten onto screen at 0.10 in the full set is mostly the pairing, with about 0.006 of it expected without screen at all, while squares near 0.01 are background.

Dropping an op also makes more contexts confident: without screen, 451 held-out lighten contexts are confident, against 78 in the full set.

The matrices also show where the HSV ops of E2 fell short. In no-multiply, confident sat-hsv and value-hsv contexts put 0.28 and 0.27 of their mass on the answers of each other, and the other runs that fell short spread smaller amounts over several ops.

Training time (E4)¶

To see whether a shorter schedule could suffice, we logged how skill on the probe set grew through training. The figure below shows each run.

The dots on each line mark where the run passed 90% and 95% of its final skill, and the table below gives those steps.

Every run reached 90% of its final skill by 74% of the way through training, and the last fifth added little (at most 0.014 of EEM). These are on the cosine schedule, where the rate falls to 1% of its peak by the end, so a shorter run would anneal sooner too and might not lose even that much. Ex-2.2.17 saw the same shape.

Discussion¶

The op set without screen, multiply, hsvmix, and exclusion is easier (its ceiling is higher) and the model gets closer to it. The higher ceiling comes from dropping ops that round at random. The smaller gap came with both four-op drops, so breaking up the similar pairs may account for it, though one run each is not enough to be sure. Dropping screen and multiply rather than lighten and darken leaves seven ops that are easier to tell apart. Adopting that set would also change the ceiling that ex-2.2.16 and ex-2.2.17 were scored against, so their results would not compare with new ones as they stand.

The single drops say less. Dropping one op mostly moved the ceiling and the model together, except in the three runs that fell short on the HSV ops; with one seed, that may be timing. The more consistent sign is that all three runs without hsvmix learned the HSV ops early, which fits ex-2.2.17 finding hsvmix the hardest op to compute.

Method¶

Corpora. Each op set has its own corpus of 300,000 contexts, its own holdout of 2,000 contexts per op, and its own probe set of 200 per op, drawn with the sampler of ex-2.2.16 from the answer table restricted to its ops. The posterior, ceiling, and floor are those of ex-2.2.16, computed on the same restricted table.

Leak. The colors a dropped op would give come from the full eleven-op answer table, so its column is defined in runs that never trained on it, and serves there as a baseline for the other columns. Confident contexts are chosen on the posterior of the op set of the run, so they differ between runs.

Cost. The scout cost about $2.42 on Modal, $2.30 of it L4 time: about $0.26 per training run. Each training run took 14 to 17 minutes on one L4, about 7,000 steps a minute.