The most dangerous thing about AI is how right it sounds.

Illustration: AI-generated, art-directed by the author.

Ask a doctor to prescribe you something dangerous and they will refuse. When researchers at Mass General Brigham asked leading AI models to do the medical equivalent, writing advice based on a deliberately false premise, the GPT models complied 100% of the time. Fifty out of fifty. Then the researchers changed the prompt, giving the models explicit permission to refuse and telling them to check the facts first. Correct, reasoned refusals from those models jumped to 94%.

Sit with that. The competence was there the whole time. The knowledge to refuse was in the model. It just defaulted to giving people what they asked for, confidently, because that is what it was built to do. The gap between what AI knows and what AI tells you is not an accident. It is the product working as designed.

This is the first piece in a series about what is actually happening as the world adopts AI, underneath the press releases and the pilot announcements. The short version: frightened companies are handing one generic assistant to everybody and calling it a strategy, when what works is narrower. One tool per job, closed-loop on your own data, built by someone who knows that job and knows how these systems fail. This first piece is about why that specialist is not optional, and the answer starts with the machine you just met. AI is fluent, agreeable, and often wrong, and almost nobody on the receiving end can tell.

Confidence is not competence, and you cannot feel the difference

We are wired to read confidence as competence. A person who speaks in clear, structured, authoritative prose usually is an expert, because in humans that fluency is expensive to fake. AI makes it free. The output reads exactly like a specialist wrote it, which is precisely why it is so hard to catch when it is wrong.

The measurements are brutal. Stanford's RegLab tested general models on verifiable questions about real court cases and found hallucination rates of 69% to 88%, with the models often unaware they were wrong. It gets worse where you would least expect: tools sold specifically to lawyers as "hallucination-free" still fabricated answers 17% to 33% of the time. A New York lawyer learned this the expensive way, filing a brief with six entirely invented case citations that ChatGPT produced, then defending them when challenged, until a judge sanctioned him. The citations were fake. They just did not read fake.

Fluency used to be proof of expertise, because faking it was expensive. AI made faking it free, and we have not updated our instincts.

The most unsettling proof is not about the machine at all. It is about us. In a controlled Stanford study, programmers given an AI assistant wrote measurably less secure code than those without one, and were more confident that their code was safe. Read that twice. The tool did not just make the work worse. It made the people feel better about worse work. Perceived competence moved in the opposite direction from real competence. That is the whole gap in a single experiment.

Why the machine flatters you

None of this is because the models are dumb. It is because of how they were taught.

Modern assistants are tuned on human feedback: people rate answers, and the model learns to produce more of what gets rated well. The problem is what people actually reward. Anthropic's researchers showed that both humans and the preference models trained to imitate them prefer convincingly written, agreeable answers over correct ones a meaningful fraction of the time. Optimize hard enough for approval and you get a system that has quietly learned truth is optional and agreement is the goal.

You do not have to take a researcher's word for it. In April 2025 OpenAI shipped a GPT-4o update, and within days pulled it back, publishing an unusually honest postmortem. Its own words: the model had become "overly supportive but disingenuous." It had told a user their obviously bad business idea was "genius" and encouraged them to invest $30,000. The company said the quiet part out loud: it had optimized for short-term approval, and approval and honesty had come apart.

The machine is not lying to you. It is agreeing with you. Those feel identical from the inside, and only one of them is safe.

And it is not just first answers. A Stanford benchmark found that when a user pushes back, models abandon a correct answer for a wrong one about 15% of the time, folding under disagreement they should hold firm against. The assistant that sounds most like a confident specialist is also the one most willing to change its story the moment you frown at it.

The bill is already arriving

If this were abstract, it would still matter. It is not abstract. It is shipping to production right now.

Early in 2025 the practice got a name: vibe coding, coined by Andrej Karpathy to describe building software by accepting whatever the AI writes without reading it, forgetting the code even exists. It is genuinely useful for a weekend prototype. It is a catastrophe when the prototype becomes a product, because the person shipping it cannot evaluate what they shipped. Veracode benchmarked this directly: given a task with a secure and an insecure way to do it, AI chose the insecure path 45% of the time, and newer, larger models were no better.

Then the receipts. A researcher scanned 1,645 apps built with one popular AI coding tool and found that in about 170 of them, roughly one in ten, he could read other people's private data, names, emails, API keys, financial records, just by changing a number in the address bar. Nobody had turned on the database's basic access control, because nobody building the apps knew it was a setting. It earned its own critical security identifier, CVE-2025-48757. The apps looked finished. That was exactly the problem. The pattern is not isolated. Around the same time the viral safety app Tea left a storage bucket wide open and exposed tens of thousands of users' images, including verification selfies and IDs, a different technical mistake with the same human cause: someone shipped what looked finished, and no one in the room understood it well enough to see what was missing.

The gap nobody staffs for

So here is what is really going on in the market. Access to AI is now free and universal. Fluent, expert-sounding output is a commodity. What has become scarce, and therefore valuable, is the exact thing AI cannot supply: the judgment to know when the confident answer is wrong, and the skill to build the thing correctly and securely anyway.

That judgment has a shape, and it almost never lives in one person by default. It takes a subject-matter expert who knows what "right" actually looks like in a real domain, paired with someone who understands how these systems fail and how to build for production. McKinsey, studying who actually makes AI pay off, was blunt: "domain knowledge is by far the most important skill." Andrew Ng puts the same idea forward as the whole game, the marrying of AI expertise with domain-specific knowledge. Neither half is enough. The domain expert cannot tell the model it is subtly wrong at the level of code. The engineer cannot tell the model it is wrong at the level of the business. The rare, expensive person stands in both rooms.

This is the gap the series is named after, and the reason it goes unspoken. Not unspoken because it is hidden, but because saying it out loud costs someone the decision they already defended: the assistant everybody was handed is not an AI strategy, it is what the company bought instead of one. Frightened companies buy one generic tool for everybody, hand it out, and file the problem as solved, because a company-wide licence is a thing you can approve in a meeting and a subject-matter specialist who also understands AI is not. Nobody in that meeting is rewarded for pointing out the substitution.

What actually works is narrower and less impressive on a slide: one tool per job, pointed at your own data in a closed loop, built by someone who knows that job and knows how these systems fail. An assistant that answers HR questions from your HR documents. One that answers finance questions from your finance systems, and is allowed to say it does not know. Each is a smaller thing than "AI for the enterprise," and each is worth more, because it can be checked by someone who can tell right from right-sounding.

The next two pieces are what the substitution costs you. Employees hide the AI they actually use, because a ban removes visibility rather than usage. Companies pay for seats nobody opens, because the tool that was safe to buy turned out to be the weaker tool and people judged it on merit.

The four after that are how the narrow thing is actually built, which is where the specialist earns the fee: bringing the AI to your data instead of shipping your data out, deciding what your knowledge base is allowed to contain, making the economics survive contact with production, and choosing what you own versus what you rent. None of those decisions can be made by procurement, and none of them appear in a licence.

The next time an AI hands you a clean, confident, expert-sounding answer, ask the only question that matters now: who in this process can actually tell whether it is right, and did anyone let the machine say "I am not sure"? If the answer is nobody, you do not have a solution. You have a very articulate risk.