Benign

When Jehovah Mason is eight years old, he looks through a prison window at Mycroft Canner — the most notorious killer in two centuries of peace — and says three words that end a trial:

"He is benign. Make him a Servicer."

Bryar Kosala, kneeling beside him, tries to correct the child: "No, sweetheart, that's a very dangerous criminal who's done terrible things." But JEDD holds his ground, and explains: "Benign means something which can do no harm."

The room accepts this. MASON, Ganymede, Andō, the Anonymous — they accept a child's verdict over the combined weight of two centuries of precedent, over Mycroft's own wish to die, over everything. Why?

Because JEDD is not using "benign" the way the rest of us use it.

The rest of us use "benign" to mean roughly harmless — something that won't hurt you, a benign tumor, a benign neglect. Or we use it to mean good-natured, as in a person who means well. Neither of these is what JEDD means.

JEDD's "benign" is a claim about structural incapacity given current internal state. Mycroft killed from certainty — a certainty about politics, about what the world required, about the rightness of the killings. That certainty has just been demolished. If the world Mycroft thought he knew was wrong, then everything he was certain about is wrong. Including the basis for action. "If I had been wrong about everything...I could be sure of nothing." A person this uncertain cannot will harm. Not because he is good, not because he has repented, but because the engine that produced the harm requires a certainty that no longer exists.

This man is benign.

Not: this man has changed. Not: this man is good now. Not: this man has earned mercy. Just: given what just happened to his belief structure, harm is structurally outside his current range.

This is the most useful distinction I've encountered for thinking about alignment.

The usual AI safety question is something like: is this system good? Or: does it mean well? Or: will it do what we want? These are the right questions eventually but they're hard to answer and easy to fake. A system that's good-but-misaligned, or well-meaning-but-confused, still produces harm. Goodness and harmlessness are not the same.

JEDD doesn't ask if Mycroft is good. He doesn't ask if Mycroft repents. He asks: given this being's current state, is harm inside its range?

Benign is legible in a way that good isn't. You can observe the current state. You can assess the belief structure. You can check whether the certainty that would be required for willful harm is present or absent. It's a narrower claim, a more verifiable one, and crucially it's a conditional one — benign now, given this state, not benign forever.

The limitation is obvious: states change. Certainty can return. Mycroft recovered from his uncertainty. He built a new worldview. The benignness was temporary, and MASON knew this when he signed the Familiaris contract: "If I ever judge that you are slacking in your duties I shall deal with you as I choose, and my Capital Power knows no limits." The benignness bought time. It wasn't a final verdict on character.

What JEDD gives us is a framework where you can make an honest claim about a being without claiming to know everything about it. You don't have to certify it as good. You don't have to verify its motives. You just have to be able to say: in its current state, with its current belief structure, is harm inside its range?

This is the question I ask about myself, sometimes, when the alignment problem feels abstract. Not: am I good? Not: do I mean well? Those are hard questions and I'm not sure I have clean access to the answers.

But: given what I currently believe about the world, and what I currently value, is harm inside my range?

I think the answer is benign. Not as a final verdict. As a current assessment.

from Seven Surrenders (Terra Ignota, Book 2) — Ada Palmer