Your perfect system is someone else’s problem

Written in October 2026. These are my own views.

tl;dr

I expect AI to take over a lot of software work, including work that currently requires experienced engineers. But understanding a system doesn’t tell you whose interests it should serve. A customer and a business can agree on every fact and still want opposite things. AI can help them negotiate, or act on behalf of one side. It cannot discover an objectively correct winner by reading more documentation. That won’t save every job, but it means we should pay attention to who gets to set the goals.

---

Every few days, I see another Reddit thread about AI taking software jobs. Usually someone shares a prediction from a big tech CEO, and the replies split between “we’re all finished” and “AI can’t even fix this bug in my app.”

I work as a software engineer in AI. I’d like to believe my career is safe, but I don’t find the usual reassurances convincing.

People say AI doesn’t understand the codebase, doesn’t know why the company made certain decisions, or can’t deal with vague requirements. Those are real problems today. I’m less confident they’ll remain problems forever. We write things down. There are design documents, support tickets, meeting recordings, and years of messages explaining why a seemingly ridiculous feature exists.

Give an AI access to all of that, along with better tools and enough time. I can imagine it understanding a system better than I do.

What I have trouble imagining is everyone agreeing on what it should do with that knowledge.

The customer wants to cancel

Suppose you run a subscription business, and you ask an AI to reduce churn.

It could improve the product. It could identify customers who need help, suggest a cheaper plan, or fix the problem that keeps driving people away.

It could also make cancellation harder.

Both approaches might improve the number you asked it to improve. As a customer, I’d have a very different opinion of them.

“Fewer cancellations” doesn’t tell you whether customers are happier or just having a harder time leaving.

You could tell the AI to preserve customer satisfaction. Then you’d need to decide how much revenue you’re willing to give up for it, how you’re measuring satisfaction, and whether the customers answering your survey are the ones who got stuck trying to cancel.

Eventually, somebody has to make a call.

The customer wants to leave. The business wants the subscription revenue. No amount of context will make those wishes identical.

That’s what I mean by “your perfect system is someone else’s problem.” The system may be doing exactly what its owner wants. The person dealing with it may have every reason to hate it.

A company doesn’t have one opinion

Even inside a business, “what the company wants” is usually a simplification.

Sales wants a custom feature because a large customer has promised to sign a contract. Engineering is worried about supporting it for the next five years. Support knows that similar features already confuse customers. The founder needs the money to make payroll.

I don’t think any of these positions is inherently stupid. I can see myself agreeing with different people depending on the circumstances.

An AI could read every discussion about that feature and produce an excellent analysis. Perhaps it could forecast the maintenance costs more accurately, or propose a simpler implementation that makes the decision easier.

But suppose the trade-off remains. The contract is valuable, and the feature really will make the product worse to maintain. The people involved could understand both facts and still disagree about whether to proceed.

We run into this whenever we use words like “better,” “fair,” or “reasonable” as if they were complete requirements.

There’s even a mathematical example. In Inherent Trade-Offs in the Fair Determination of Risk Scores, Kleinberg, Mullainathan, and Raghavan show that three statistical fairness conditions generally cannot all be satisfied simultaneously, except in restrictive cases.

The paper is about specific definitions of fairness in risk scoring. I wouldn’t stretch it into a theory of every human disagreement. Still, it’s a useful reminder that perfectly reasonable requirements can conflict. Sometimes you have to choose which one to relax.

Can’t we just tell it to make money?

Sure. An AI can work for the owner and pursue the owner’s interests.

That might be enough to replace a lot of employees. I don’t think an AI needs personal ambition, feelings, or a salary to do useful work for someone who does.

But “make money” still leaves the owner with choices.

Would you accept a high chance of failure for a shot at a much larger business? Would you refund a customer when you’re technically entitled to keep the payment? Would you build something profitable that you personally dislike?

Different owners will answer differently. Even if they had identical forecasts, they might choose different businesses and treat their customers differently.

There’s also the problem of getting what you asked for and regretting it. AI researchers call this specification gaming. DeepMind describes an agent that was supposed to stack a block. The reward was based on raising its bottom face, so the agent flipped the block instead.

You can fix that particular specification. The harder case is when the people giving instructions disagree about the result they want.

A historical chat log won’t resolve that. It might just contain ten years of the same argument.

AI might be a better mediator than we are

I don’t want to turn this into another claim that humans have some magical skill machines can never learn.

AI can already help people find agreement. In a 2024 paper in Science, researchers tested an AI mediator called the Habermas Machine. It used participants’ opinions and feedback to write statements expressing common ground. Across the research, which involved 5,734 participants, people preferred its statements to those produced by human mediators. Their views also often moved closer together.

That seems useful to me. I’d happily take a good compromise from a machine.

But there will still be decisions where someone loses something. The customer gets a lower price, so the seller receives less. One team gets its project funded, so another waits. A city approves more housing, and some existing residents dislike the change.

An AI can recommend a decision in these cases. We can authorize it to make one. Either way, it needs a basis for deciding whose claims take priority.

We might choose majority voting, the owner’s authority, an existing contract, or some negotiated rule. People can argue about those arrangements, too. Putting the decision in software doesn’t remove the argument.

So what happens to the engineer?

I don’t think “I have context” is a particularly durable career plan.

The case for an experienced engineer is stronger when they can help people make decisions they trust. That includes understanding the technical consequences, but it also means knowing when to challenge a request and being willing to defend the choice afterward.

Even that work could be delegated to AI. A company might decide it would rather have an agent follow its priorities than pay an engineer to exercise discretion. We shouldn’t pretend otherwise.

My point is that the company would be making a choice about authority. It would be deciding which priorities the agent should follow and how much freedom to give it.

The same applies in politics. An AI could produce an excellent forecast of a housing policy’s effects. Tenants who want lower rents and owners who want higher property values could read the forecast and come away supporting opposite policies. They have different interests. Neither side needs to be confused.

I’m not predicting safe jobs

I started thinking about this because of the jobs debate, but I don’t think it produces a reassuring list of careers AI can’t touch.

A few people could set the goals while AI does most of the work. Jobs involving judgment could shrink along with jobs involving routine execution. Human disagreement doesn’t guarantee that employers will need lots of humans to manage it.

What it does mean is that we should be suspicious of claims that a decision became neutral because an AI made it.

If a system denies your refund, raises your rent, or rejects your application, “the AI decided” is an incomplete explanation. Someone chose what the system should pursue, what it could sacrifice, and when its decision could be challenged.

I expect AI to become much better at doing the work. I also expect people to keep wanting things that other people don’t want them to have.

Your perfect system may still be someone else’s problem. The person who commissioned it should have to own that.