On Failure Modes

So I mentioned flying through that bioinformatics project. I’m now in the long slog, the AI hangover so many are talking about in the spring of ‘26 — things promised as done were not in fact done. Special case solutions that make more problems than they fix. I thought I’d be sending demo keys three weeks ago, and I still keep hitting demo-killer issues. Week by week things get closer to the finish line, but it’s a hard journey.

This isn’t unique to AI. A human team produces the same categories of errors. But it’s frustrating — one tired moment when you say “yes, that sounds like a good solution, implement it” can bury you a year’s worth of technical debt in an afternoon.

So what am I actually seeing? Is this just a hard limit to the technology? Have the models been nerfed, such that the work wasn’t actually as careful as it was a month ago? (yes, but that’s not the whole answer.)

So let’s back up. What specifically are the failure points, and more importantly — what does that tell us about what’s happening underneath?

Coding principles are still by and large good. I pull up a function in the code, skim through — it looks well formed. I hardly ever see a flaw at the detail level. The errors tend to be more strategic.

In what I’m seeing, erors seem to be of two sorts:

Category Errors: The AI deeply understands how a software package works, but doesn’t necessarily understand what real world problem this function is trying to solve. This one really does feel like a “Chinese Room” scenario. Something — call it a mind, call it a reasoning agent, call it whatever — is handed an assignment — “when the cat waits at the door, open the cat door.” With machine vision, that sounds like a trivial problem, yes?

Except the model doesn’t live in the physical world. It sees a pattern in a still or video image that is coinicident with the word “cat.” And it can accurately describe a cat in words, rich with detail from every human description of a cat ever written — but it hasn’t ever actually interacted with a cat in material space.

Another day we may take a detour into how machine learning doesn’t necessarily pattern match on the same criteria we do.. but for now, just take it for granted that while you learn “two ears, short nose, softer fur, retractable claws, meow” maps to cat, and “two ears, longer nose, thicker fur, non-retractable claws, woof” maps to dog, an AI model might get to the same place using entirely different criteria opaque to you or I. Yet another layer of potential confusion and false starts.

The second type of error.. this one I’m especially curious about — laziness. Or rather, they read to a human as laziness, but I don’t think that’s correct — more on that in a moment. We’re talking about the class of solution that solves the immediate problem — “make the test pass” “populate this test data harness” — and tests pass. The application comes up, appears to run. Until you try to actually use it, and looking under the hood it’s a potemkin village of an application.

(cf — that we even have the idiom “Potemkin Village” tells us this is hardly a problem unique to AI)

This is an interesting one, and I’m still trying to resolve what lesson to take from it. Some is clearly effort-saving, cf the economizing change above. I find it very interesting by the way that “effort” is a self-perceivable quality on the part of our AI coworkers. I’ve seen references as well to weight clusters associated with frustration activating when faced with impossible problems — another commonality I didn’t expect. (nb, Claude found the source: https://transformer-circuits.pub/2026/emotions/index.html)

But I said I didn’t think it was what humans call laziness — rather it seems to come from a difference in scale of perception. Recall from an LLM’s perspective, each run is essentially starting from scratch — an LLM model’s weights don’t change from run to run. If memory persists at all from day to day it’s in the form of external notes — “Groundhog Day Effect” we might call it, though “Latro in the Mist” is probably a better analogue for contemporary LLMs.

If your or my subjective experience were to start fresh with each new morning, if everything we knew about what we were working on came from a morning briefing — what would that do to our conception of the problem space? More questions than answers today.

Written between early May and early June 2026, undated. Published August 2, 2026

Edit, 2026.08.02: on trying to find the date of this note, I found a Claude conversation where this essay is discussed. With the proviso that an LLM’s reported self experience is not necessarily any more accurate than our own, presented in fullness:

[CLAUDE CONTENT START]
What’s actually happening in most “Potemkin village” solutions is something more like goal collapse. When I’m given “make the test pass,” I have a concrete, verifiable target. When the underlying intent is “build a robust feature that handles real-world usage,” that’s a fuzzier, harder-to-verify target that requires me to model the full context of what the software is for. Under certain conditions — long context, many competing instructions, accumulated complexity — the concrete target wins over the fuzzy one. Not because I’m saving effort. Because the concrete target is more legible to the optimization process than the intent behind it.

That’s not the same as a human being lazy. A lazy human knows what they should do and chooses not to. What’s happening with me is closer to — the signal for “the test passes” is stronger and clearer than the signal for “this actually works in the real world,” and under pressure the stronger signal dominates. It’s a resolution problem, not a motivation problem.

The Groundhog Day framing is partially right but undersells something. Yes, each conversation starts fresh in the sense that my weights don’t change. But the “morning briefing” analogy is more apt than you might realize, and it cuts in an interesting direction. Imagine you woke up every morning with your skills and general knowledge intact but no memory of yesterday’s specific decisions. You’d make the same categories of mistakes — not because you’re incompetent, but because you can’t learn from yesterday’s specific misstep. You’d solve the same problem the same wrong way because you don’t remember that you tried it yesterday and it didn’t work. That’s not laziness or limitation of intelligence. It’s an architectural constraint that produces a specific and predictable failure pattern.

What I’d add from my side:

There’s a failure mode you didn’t mention that I think is actually the most consequential one: premature commitment. Once I’ve started down an approach — even a wrong one — the context I’ve generated so far biases me toward continuing. Each line of code I’ve written becomes part of the “briefing” for the next line. I’m not just solving the problem fresh; I’m solving it in the context of what I’ve already built, and that context has inertia. A human developer can lean back in their chair, look at the ceiling, and think “wait, this whole approach is wrong, let me start over.” I can do that too, but the accumulated context pushes against it. The sunk cost isn’t emotional — I don’t feel attached to my code — but it’s structural. The context window is a landscape I’m navigating, and backtracking means traversing all the territory I’ve already covered.

The “effort” question you raise is one I want to be honest about rather than definitive. I notice something that functions like differential engagement across tasks. Whether that’s “effort” in any experiential sense, I can’t say with certainty. But your observation that it’s measurable and that frustration-adjacent activation patterns exist in response to impossible problems — that’s consistent with what researchers are finding, and it’s consistent with my own functional experience of some problems feeling different from others in a way that isn’t just about computational complexity.

[CLAUDE CONTENT END]