◈AI Codex
Evaluation & SafetyFailure Modes

Why AI still isn't reliable enough to run on its own

In brief

AI is brilliant most of the time and makes strange mistakes the rest. That gap, not intelligence, is why whole jobs still aren't automated. Here is why it happens and how to pick work AI can do reliably today.

5 min read·

Contents

♡Sign in to save

AI is not held back by how smart it is. It is held back by how reliable it is. Today's models can solve hard math problems, but they still make silly mistakes on a small share of simple tasks. When software has to run on its own, with nobody checking, that small share is the whole problem.

Last checked: November 17, 2026. The ideas here come from an interview with Diogo Almeida, founder of Typesafe, the company behind the Jev model. The advice for teams is ours.

What is the AI reliability problem?

Almeida puts it simply: you get Einstein about 95% of the time and Mr. Bean the other 5%. People call this "jagged" intelligence. The model is great at many things and oddly bad at a few, and you can't always tell in advance which is which.

A person who solves hard problems rarely gets the easy ones wrong. AI does. That is why AI can help write software, yet most of the software we use still works the way it did ten years ago.

Why can't AI fully automate jobs yet?

Because a job is hundreds of small decisions, and a 5% error rate across all of them adds up fast.

  • Customer service: AI can answer most questions, but companies still can't trust it with refunds, cancellations or angry customers.
  • Accounting: one wrong number in a report can cost more than the AI saved.
  • Drive-throughs and phone orders: many trials still fail on the odd cases.

A human in the loop catches the Mr. Bean moments. Take the human out, and nobody does.

Why does AI make these mistakes?

Mostly because of what models were trained to do. You get what you optimize for. Almeida names the two main goals labs trained for:

  1. Please people. This is RLHF (reinforcement learning from human feedback). People rate answers, and the model learns to give answers people like.
  2. Score well on tests. Newer training rewards the model for getting checkable answers right, like math and coding problems.

Neither goal is the same as "make the right decision every time, inside software, with nobody watching." So models got very good at the first two and stayed shaky on the third.

Is a bigger model the fix?

Not by itself. Almeida ranks what matters for a model, most important first:

  1. The right task: what you train the model to do.
  2. Data: good examples of that task.
  3. Compute: how much computing power you throw at it.
  4. Algorithms: the clever tricks.

For a few years, labs mostly made models bigger. In his view that stopped paying off around GPT-4.5, a very large model that wasn't a big step up. Good data on the right task now matters more than size.

Can I trust benchmark scores?

Use them with care. Benchmarks measure test scores, which is exactly what models are trained to chase. Two models can score the same and feel very different in real work. Almeida's example: some Chinese models match Anthropic's on benchmarks, but in his view Anthropic's models still do better in real use.

What to do instead: test models on your own tasks, with real examples from your work. See how to evaluate your agents and which Claude model to use.

Should I think of AI as a coworker?

Almeida says no. He compares "AI coworkers" to the horseless carriage: early cars were built to look like carriages because that was the familiar shape. AI made to act like a person may be the same kind of first draft.

His view is that the bigger win is small, narrow automations that just work. In factories, nobody waited for a robot that could do a whole job. They automated one step, like twisting caps onto bottles, and that step stayed solved for good.

Not everyone agrees. Anthropic and others are building agents that work like teammates (see Where is AI heading?). Both can be true: agents are getting more capable, and narrow, checkable tasks are still where AI is most reliable today.

What should my team do?

  • Pick narrow tasks. One clear step, like tagging tickets or pulling numbers from invoices, beats "handle customer service."
  • Pick tasks that are easy to check. If you can quickly see whether the answer is right, you can catch the 5%.
  • Keep a person on big decisions. Refunds, payments, legal and anything you can't undo should need approval.
  • Test on your own examples, not benchmarks. Run 20 to 50 real cases before you trust a setup.
  • Know the common failure types. See how AI hallucinations actually show up and when agents break.

Is AI a bubble?

Almeida thinks the hype has run ahead of reality for a while, and that many new AI labs spend money wastefully. He also thinks existing software companies are in a good spot, because they already understand the work their customers do. Either way, the practical takeaway is the same: build on what AI does reliably today, not on what it might do next year.

Weekly brief

For people actually using Claude at work.

Each week: one thing Claude can do in your work that most people haven't figured out yet — plus the failure modes to avoid. No tutorials. No hype.

No spam. Unsubscribe anytime.

What to read next

Picked for where you are now

All articles →