How far the answer moves: an RGB-distance readout beside expected exact match¶

Observations¶

• The removal lines, by distance: under projection, the greedy guess on handover moves from the floor (0 to 0.5 steps, a perfect answer) to 1.8 steps on mix and 3.4 on value-hsv, between 44% and 77% of the way to chance. The control stays at the floor. The mean, median, and expected distances agree with the greedy one within 0.3 steps.
• Kept and lost: the operator loses 77% to 99% of the removal lines. The kept lines stay at the floor; the lost lines sit 1.8 (mix) to 3.7 (value-hsv) steps from the answer, and reach chance only on value-hsv.
• The lost answers on the cube: on mix, nine in ten lost answers are one or two steps off. On hue-hsv the guesses keep their saturation and move in hue; on darken they land darker than the answer.
• A counterfactual answer: the lost answers are not the op applied to a de-reddened operand. The nearest such answer is a little closer to the guess than the true one on most ops, but the guess lands in its box on under a quarter of the lines on all ops but lighten.
• The non-red lines: on handover the exact match deficit stays under 0.01 and the mean of the distribution moves under 0.03 steps on every op. The control's own lines move as much or more under the same operator.
• Between seeds: the seed spread of the distances is 6% to 20% of the mean, against 33% to 127% for the kept share of exact match.

Scope¶

This is a re-score of stored checkpoints, planned in the backlog item. There is no training, no gates, and no verdicts.

Every removal and selectivity number in D2.2 is expected exact match: the probability mass the model puts on the colors that a line's answer can be. It gives no partial credit, so a guess one grid step from the answer scores the same as a guess of black.

So when handover keeps 24% of exact match on the hue-hsv removal lines, that could mean a quarter of the lines answered perfectly and the rest sent anywhere, or every line nudged one step over. Likewise, a non-red deficit of 0.01 could be one line in a hundred lost outright, or the mass of every line leaning a little.

We add a distance on the color grid, in grid steps. One step is the spacing between the six levels of each channel, and the far corner of the cube is 5√3 ≈ 8.7 steps from black. The target of a line is its raw answer (the unrounded value of the op), which is also the expected value of the answer distribution under stochastic rounding.¹

Four quantities are measured from the raw answer. Three are central points of the model's distribution over the 216 grid colors: the greedy guess (the most likely color), the mean, and the channel-wise median. The fourth is the expected distance of a color drawn from the distribution.

The greedy and expected distances are the ones to compare with exact match. The mean is the one that moves when mass leans toward a neighbor while the guess stays the same.

Every distance comes with two references. The floor is the distance from the raw answer to the best possible grid answer (the mode of the answer distribution): zero when the answer lands on the grid, and up to half a step per channel otherwise. A distance at the floor is a perfect answer.

Chance is the mean distance of a color drawn uniformly from the grid: about 3 steps for a target near the middle of the cube, and more for one near a corner. A distance at chance means the answer no longer depends on the line.

The distances and exact match are measured under the same three passes as ex-2.2.11: (a) the clean forward pass, (b) projection, with e₁ removed at every slice and position, and (c) operands, with e₁ removed at the operand positions only. Each is read on the line groups of ex-2.2.11: the removal lines (a red operand whose hue the op carries into the answer), the non-red lines, and the rest. As in the ex-2.2.11 report, the shaped operator is left out.

The runs¶

The runs are the 54 final checkpoints of ex-2.2.11: the un-anchored control (5 seeds), handover (20), and two arms that each undo one of the changes in handover, handover-slot (20) and handover-tied (9). The probe lines are the ones ex-2.2.11 scored: 5,832 per op, and twice that for the three order-sensitive ops.

The two scorers agree: the re-score reproduces ex-2.2.11's kept share of expected exact match on every op (24% on hue-hsv, 12% to 18% on the next four).

The removal lines, by distance¶

On the removal lines under projection, the greedy guess on handover moves about halfway from the floor to chance on six ops (44% to 51% of the way), and further on difference, exclusion, and the three HSV ops (61% to 77%). The figure puts the distance beside the kept share, per op.

The clean pass sits at the floor on every op and condition (the table below the figure gives it), so any distance above the floor under projection is caused by the operator.

A sampled answer lands about as far as the greedy one. Sampling at temperature 1 draws from the model's distribution, so the expected distance is what it scores: 0.03 to 0.18 steps above the greedy distance on every op (the table gives it). The greedy guess is the limit as the temperature goes to zero. The mean of the distribution describes no single sample; it is the summary that moves when mass leans toward a neighbor while the guess stays put, and it sits a little below the greedy distance.

The table gives the same measurements on handover, adds the clean and operands passes, and adds the hit rate (the share of lines whose greedy guess is one of the line's possible answers).

Kept and lost¶

A distance averaged over a group mixes lines the operator leaves answered with lines it does not, so we split the lines by the greedy guess. Under a given pass, a line is: (a) kept when the guess is one of the possible answers of the line, (b) lost when the clean guess was a possible answer and the guess under the pass is not, and (c) never when the clean guess already missed. The distance of the lost lines tells us where a removed answer goes.

Per line, as a distribution: in the histograms below, the bin at zero holds the kept lines and the other bins show where the lost answers go.

On mix, nine in ten of the moved answers sit one or two steps off, which is what treating the red channel as absent would give (on mix that shift is about 2.5 steps). darken decays from one step, hue-hsv spreads over two to four, and value-hsv is close to the shape a uniform guess gives.

The lost answers on the cube¶

The figure draws the lost removal lines of the two ops the backlog item names: hue-hsv, whose removal lines keep the most exact match, and darken, the near miss of ex-2.2.11.

A counterfactual answer¶

This is a post hoc check on where the lost answers go. If the operator took the redness out of the red operand and left the op intact, the guess would be the op applied to a de-reddened operand. We try two versions of that operand: (a) red channel set to zero, and (b) red channel lowered to the larger of its green and blue, which keeps the other two channels and drops the saturation.

Neither substitute is where the guesses go. The zero-red answer is farther from the guess than the true answer on every op but lighten and hsvmix. The grey-red answer is a little nearer on most ops (2.0 steps against 2.5 on hue-hsv), but the guess lands in its box on under a quarter of the lines on every op but lighten (36%).

So the lost answers sit between the answer and its de-reddened version, and no single substitute operand describes them.

The non-red lines¶

The non-red lines are the other side of selectivity: the operator should leave them alone. Ex-2.2.11 gates the exact match deficit on mix at 0.05, and finds it near zero on every op. The distance shows whether the mass moves at all on those lines.

Between seeds¶

The distances vary far less between seeds than exact match does. A measurement used for a gate needs a seed spread small relative to the effect it measures; the table gives each removal measurement's spread across the twenty handover seeds as a share of its mean (the coefficient of variation, CV).

What we make of it¶

Under projection, nearly every removal line moves, and moves part of the way: the greedy guess ends about halfway from a perfect answer to a random guess on six of the eleven ops, and further on the rest. On mix, the op the D2.2 gates are scored on, the guess ends one or two steps from the answer nine times in ten.

The answers move toward what the line would give without its red, but not all the way: the counterfactual check says the model is not computing the op on a de-reddened operand.

That is the kind of side-effect this program wants to be able to describe. The intervention changes the answer on the lines that use the concept, by an amount that scales with how much the op relies on the red channel, and leaves the other lines alone.

The distance gives the removal measurement partial credit and a scale. On mix, exact match on handover sits near zero with a seed spread larger than its mean, while the distance on the same lines has a seed spread under a tenth of its mean.

That comparison flatters the distance: the CV favors a measurement with a large mean, and the distances have means an order of magnitude above the kept shares. In units of the seed spread, the two separate the conditions about equally well. The advantage of the distance is that it stays informative where exact match is near zero.

The distance also separates ops that exact match does not: mix and value-hsv both keep under a tenth of their exact match, but their guesses sit 1.8 and 3.4 steps away.

On the non-red lines, the mean of the distribution is the sensitive measurement, since it moves while the guess stays put. On handover every such move is under 0.03 steps, and the control's lines move as much or more, so the operator's cost on the lines it should leave alone is about what removing any direction costs.

We adopt the distance for D2.2's removal and selectivity gates, from the next experiment that scores them, with this direction: (a) removal, as the normalized distance of the mean of the distribution on the removal lines, (distance − floor) / (chance − floor), rising under the operator, with the greedy distance beside it as the number a user of exact match would expect; and (b) selectivity, as the non-red rise of the same mean staying at or under the rise of the control under the same operator. The thresholds are the one thing a re-score cannot supply, since it has no gate to calibrate against, so that experiment sets them from its own control runs. Until one of the distances has been used in a gate, the greedy and mean distances are quoted together.

The distance does not say whether an answer is right, except at the floor, so a gate that wants "answered" should keep the hit rate or exact match beside it.

Method¶

The distances are in sca.answer_distance, and the re-score in experiment.py beside this script. Each of the 54 final checkpoints of ex-2.2.11 is run over the same probe lines under the clean pass, projection, and operands, with one compiled forward pass per operator. The answer distribution at the pre-answer position is restricted to the 216 color tokens.

For each line, we compute against its raw answer: the greedy guess, the mean and channel-wise median of that distribution, its expected exact match and expected distance, and whether the guess is a hit. For each group and pass, these are averaged, split by kept, lost, and never, and binned by rounded greedy distance. On handover and the control, the per-line guess is published for every op, and the mean beside it for hue-hsv and darken; every other per-line array stays in the run's memo record.