Human Compatible
Stuart Russell argues the standard AI paradigm of fixed-objective optimization is dangerous, proposing uncertain preference models as a technical safety solution.
I keep coming back to this one when people ask me where to start with AI safety. Russell has to specify objective functions for a living. He co-wrote the textbook half of academic AI learns from. So when he says the field’s standard approach, pick a fixed goal and optimize it as hard as possible, is dangerously wrong, he’s not doing philosophy from the outside. He’s describing a bug in his own toolkit.
The argument goes like this. Give a system a fixed goal and make it good at pursuing that goal, and it will treat the goal as correct by definition. It has every reason to resist being shut off, corrected, or retrained, not out of malice, just because any of those things get in the way of the goal it already has. You don’t need a robot uprising for this to be a problem. You just need an optimizer that’s good at optimizing.
His fix is the interesting part, and it’s a real research program, not a slogan. Build machines that are trying to satisfy human preferences, but keep those preferences explicitly uncertain in the machine’s model of the world. An uncertain machine has a reason to defer, to ask before it acts, to let itself be corrected. Not because you told it to. Because deference is the smart move when you don’t know what you’re supposed to want. He calls it cooperative inverse reinforcement learning, and it’s a clever way to make safety fall out of the objective instead of getting bolted on afterward.
Where it shows its age: this was written before anything resembling a transformer existed. There’s nothing here about in-context scheming or sycophancy or the specific ways agents fail when you give them tools. For that you have to read the lab papers directly. Anthropic and OpenAI have both published on models faking alignment during training, which is a different flavor of problem than anything in this book. But RLHF itself is basically a crude, industrial-scale attempt at the thing Russell described in theory: learn what humans want from their feedback. He named the shape of the solution before anyone built a rough version of it.
Here’s what I sat with, reading it again for this. I lean hard on the idea that alignment is a property of governance, not the model. Russell’s book is a direct challenge to that as a complete claim. You can get the incentives right, get the institutions right, regulate the hell out of who’s allowed to build what, and you still have an open technical problem underneath: how do you build something that treats its own goal as up for question rather than fixed. Governance decides who builds and under what conditions. It doesn’t solve the control problem. I don’t love conceding that. The book earns it anyway.
Recommended for anyone doing technical AI work who hasn’t sat with the foundational safety arguments, and for anyone in policy who wants to see what’s underneath the compliance layer everyone’s currently arguing about. The examples are dated. The core claim isn’t.