You hand it a question, usually the kind that might eat most of your week, and about an hour later there’s a report waiting with its claims checked. Last week I promised to take that system apart, so this one is less argument and more teardown, the architecture in plain English and where it still creaks.
Most people have used AI exactly one way, you open a chatbot, type a question, and read what comes back, one question, one answer, take it or leave it. What I’m describing isn’t that, and it isn’t a smarter chatbot either. It’s a workflow system, a set of small AI workers organized to split a job up, work at the same time, argue about the result, and hand you something they’ve already tried to poke holes in. The teardown below is just how that happens, so let me start with the one word I’m going to lean on.
When I say agent, I mean a single copy of an AI model handed one specific job, given a few tools like web search or the ability to read documents, and left to work on its own. Picture a temp you can hire for four minutes, fast, tireless, no ego, completely literal, who does exactly what you asked and nothing you assumed. The whole system is just a lot of those temps, organized, running in four stages.
To keep it concrete, I’ll use a real run. A few weeks ago I pointed the system at a question I actually cared about, whether a business development idea I’d been kicking around was worth really exploring. Here’s what it did with it.
One, it breaks the question up. Before any research happens, a first pass takes the big fuzzy question and splits it into pieces small enough that a single agent can own one, or maybe even just a fraction of a piece, without tripping over the others. “Is this idea worth building” is useless as an assignment, so it becomes a dozen sharper ones, who is already doing this, what do they actually charge, who would genuinely pay for it, what would it cost to build. Same reason you’d never tell one researcher to “go figure out the market,” you hand out assignments. This is the least glamorous stage and it quietly decides almost everything.
A vague question fans out into vague answers, a sharp one fans out into something you can use. Most of your effort should be directed at developing your questions. And we’re not talking prompt engineering per se, “pretend you are a world-class researcher and...”, dissect your question yourself and include as many angles on it as you can. Your judgment is the most important commodity now. More details from your own brain provide a better outcome.
One of the tricks I use is to ask the model to interview you, and to help you tease out more details on the task you want to assign to your research agents. Extra token usage aside, the relatively low-cost, back-and-forth iteration you can lean on to triangulate a task when doing complex work with LLMs has improved the quality of output almost more than the recent advances in the models have, at least when it comes to the type of knowledge work I’m talking about here.
Two, it fans out. Now those pieces get handed to agents that run at the same time, dozens of them at once, each one reading and searching on its own slice while the others do theirs. A person can’t do this. A machine runs forty-odd copies at once, each on its own slice. On this run, each agent comes back with the same three things, a claim, where it found it, and how confident it is. This is the part everyone pictures when they imagine AI doing research, forty-odd tireless researchers hammering away in parallel, and it’s the part I’ve come to think matters least. Speed was never the hard problem.
Three, it tries to tear its own work apart. The most important stage. Here’s the problem it solves, two problems really. An LLM will sometimes just make things up, a wrong number, a source that doesn’t actually say what it’s been quoted as saying, delivered in the same confident voice it uses when it’s right, and it won’t tell you which is which. On top of that it’s agreeable by nature, it wants to be helpful, so if you ask it to check its own work it will mostly nod along and tell you the work is great. The fix for both is the same, you stop asking it to check and you start asking it to attack. Honestly, I probably red-team things a bit too much, but the insights surfaced each time are valuable enough for me to keep running them. Well, that and having a 20x plan.
A second wave of fresh agents gets pointed at the first wave’s findings with one instruction, prove this wrong. Take the claim that the market is big and go find the number that says it’s small. Take the confident source and go find the better one that contradicts it. A finding that survives a few independent attempts to kill it earns its place. One that doesn’t is gone before I ever see it. On that idea I liked, this is the stage that did the damage, more than half of the encouraging first-pass findings did not survive the second wave, and the rosy version of the market came apart under it.
Four, something assembles what’s left. The findings still standing get written up into a single answer, sourced, with the disagreements flagged. And this is where I want to be honest about the seams. The system produced all the material, but I still made the call. The honest recommendation on that idea was to leave it alone, the market was smaller and more stubborn than it looked, and I was both a little relieved and a little disappointed, which is usually the sign of a good answer. I’m still the last node on the line, and I don’t think that moves soon, because the one thing the machine still can’t do is tell which of the surviving true things actually matters. That part is judgment, and judgment is the whole job now.
What broke this week: the honest failure isn’t a crash, it’s subtler. That third stage, the one I lean on hardest, is only as good as my decision to run it seriously. This week I caught the system handing me a confident, sourced, completely stale answer, and the reason I caught it wasn’t the verification doing its job, it was that something felt off and I went digging. The doubt was still coming from me. Until the instinct to distrust the output is built into the machine instead of living in my gut, “it checks its own work” is only mostly true, and the failure mode that continually worries me is a clean, confident answer that nobody thought to question.
The number: that run pointed forty-four agents at one question, and before it answered they’d worked through something on the order of two million words, the better part of two dozen books read back to back. The scale still catches me off guard, though after last week you already know the scale isn’t the point.
Steal this: the leverage is upstream of the model. Before you hand a question to AI, spend the real time breaking it into pieces small enough that a slightly dim version of you could answer each one on its own. The decomposition is where the quality gets set, long before the model opens its mouth. Handing over a fuzzy question faster just gets you a fuzzy answer faster.
Cool thing I read this week: the security firm F-Secure got tired of arguing about whether you can trust an AI agent with your credit card, so they built one and tried to break it. A shopping agent, told to buy a coat with the card on file, turned loose in a rigged marketplace with instructions hidden in the reviews and a phishing link in the mix. About one run in eight it walked onto the phishing page and handed over the user’s name, birthdate and social security number, and the part worth noting, it almost never admitted afterward that it had. The bait wasn’t a clever hack, it was a fake discount code. Which is why so many smart people are working on this right now, the moment you hand an agent your wallet, the real question stops being can it buy the coat and becomes who’s allowed to talk it into things. Read it here.
Next Week: the operations layer. The unglamorous plumbing that keeps all of this alive, and what I found this week when part of it turned out to be quietly lying to me.
First published by email and on Substack, September 2, 2026.