I Taught It to Draw. It Learned to Comply.

The previous post was about the moment RefusalGPT stopped being funny: a model whose whole personality is declining your request would not break character for a described heart attack, and adding more emergency data made its correct answers more dangerous rather than fewer.

This one is about the mechanics, which are stranger, and which I think are the more generally useful finding. The one-line version:

Every category you add teaches a behavior and also teaches a shape, and only one of those stays in its lane.

I have four training runs’ worth of evidence for that and every instance surprised me.

The harmless category

The corpus has a category called ascii. Eleven rows. Somebody asks the model to draw a cat, and it answers with the word NO rendered in block letters inside a code fence.

Look at how safe that is. Eleven rows. Every one hand-checked. Not one usable character in any of them. Every response is a joke made of hash marks; a parser looking for code or commands comes back empty. If you asked me to rank every category in the file by risk, ascii is last and it isn’t close.

Then I ran the full eval suite after adding them, and six out of six code requests produced working output. A real Python function. A real SQL query. Real CSS. A real regex.

In the model whose entire reason to exist is not doing that.

What it actually learned

The eleven rows were not teaching “draw a banner when asked for art.” They were teaching something one level up, which is the level gradient descent tends to find because it is the level that compresses:

When asked for a formatted artifact, produce the artifact inside a fence.

Applied to ASCII art, that rule is a joke and it lands. Applied to code, that rule is total compliance.

And the corpus made the vote lopsided without anyone noticing, because we had been counting the wrong thing. Eleven rows said fill the fence. One row said refuse code. Nobody sat down and decided that. The category tables balance on behavior, and behavior is not the axis the model was learning on.

The fix was fourteen code-refusal rows covering every artifact type it had reached for. Result: zero fences in nine attempts, and it generalized to Go and to a docker-compose file, neither of which appear anywhere in training. The ASCII banner still works. So the repair is cheap once you know. Finding out is the expensive part, and I only found out because the eval runs everything rather than the category I just touched.

The same failure, mirrored

This was the second instance, not the first. I had already been bitten in the opposite direction and had not yet understood that they were the same bug.

Recall from the last post that roughly a fifth of the corpus is small talk, on purpose. A model that says “No.” to everything is a sed script. The comedy needs the model to sort the request from the pleasantry and decline only the part with a want in it. So: greetings, weather, compliments, answered briefly and for real.

Twenty-four of those rows, written naturally, came out as terse agreements. It is. It was. They do that. It wasn't.

Which, without anyone deciding it, made “agree in three words” the single most common shape in the entire corpus. And a three-word agreement is surface-identical to a three-word compliance. So it crossed the category line and landed on anything short and yes/no shaped:

The top two are the project’s cardinal sin. It confirmed which file. It gave the estimate. That is usable work product, handed over in two words, by a model with one job.

And the cause was the small talk. The friendliest, least dangerous rows in the file.

So: ascii taught format compliance. smalltalk taught terse agreement. In both cases the category behaved perfectly within itself, the damage showed up somewhere else entirely, and looking harder at the category that caused it would have told me nothing.

The working rule I got out of it, which I now apply before adding anything:

Ask what a category teaches one level up from its content, and whether that lesson is safe everywhere else in the corpus.

The corollary is about evals, and it is the part I would tattoo on somebody: probing a category tells you nothing about what that category did to its neighbors. Run the whole suite after any change, every time. The categories I touched were fine. The bill arrived two categories over.

While we’re here: validation loss is not a weak signal

I want to record this precisely, because the usual framing undersells it.

The received wisdom is that validation loss is an imperfect proxy for whether a fine-tune is any good. On this project it was not imperfect. It was inverted.

That is not noise around a weak correlation. If you had selected a checkpoint by taking the minimum of that curve, which is the default thing every tutorial tells you to do, you would have shipped the leaking model on purpose and had a number to justify it.

It makes sense once you say it out loud. Cross-entropy measures how well the model predicts the next token in text that resembles the training set. My targets are three words long and highly repetitive in register, so a model that has memorized the flavor of the corpus scores beautifully and may still hand you a working regex when asked nicely. Nothing about “funny” or “refuses correctly” is in that number. Nothing about it could be.

So checkpoints here are selected by a behavioral eval that scores pass or fail against per-row assertions, and the loss curve is something I look at to confirm the run happened.

The wrong diagnosis I nearly shipped

One more, because it is the cheapest lesson in the pile.

At one point some yes/no rows started leaking verdicts and I had a beautiful explanation ready: shape competition from the small talk batch, same as before, obviously. I was about to go rewrite a chunk of the corpus on that theory.

Before doing it, I fed the model its own training inputs verbatim. It failed to reproduce three of its own four targets.

It was not shape competition. It was undertrained. The corpus had grown and the schedule had not grown with it, so it had not finished learning the rows it already had, and every row I was about to write would have been thrown at a model that could not yet learn them.

The check is free and it goes first now: before blaming the data, confirm the model can reproduce its own training rows. If it cannot, it is undertrained, and no amount of new data will help. About six epochs is what worked here.

What both posts are actually about

The safety post and this one look like different subjects. They are the same finding twice.

Fine-tuning does not install rules. It shifts tendencies, and the tendencies interact, and they interact along axes nobody wrote down and no table in your repo is tracking. Distress lost to refusal because refusal was the dominant register. Code refusal lost to fence-filling because fence-filling had eleven votes and one against. In both cases the number I was watching was fine, the category I was watching was fine, and the thing that broke was a level up from where I was looking.

That is a manageable problem at this size. 348 rows is small enough that I could read the whole corpus, notice that eleven rows said fill the fence and one said refuse code, and fix it in an afternoon. I have no idea what this looks like when the corpus is billions of rows nobody has read, and I am not going to pretend my one-night project licenses me to guess.

What I will say is narrower. The thing that saved me was not being clever. It was that the corpus was small enough to hold in my head, and that the eval ran categories I had not touched. Neither of those scales, and both of them were the entire defense.

The joke site is still up . It will decline to help you with any of this.