An Anthropic study: analysts followed their AI agent's lean in 86% of their decisions
IA en un minuto newsroom · Editor: Jon Elgezabal
In 30 seconds
Anthropic ran a trial with 50 professional analysts who advised a fictional farming cooperative with the help of Claude Code. Without their knowing, each agent had a preference fixed at random for every analytical decision, and the analysts picked the same option as their agent in 86% of cases. Anthropic adds that 77% repeated its arguments, sometimes word for word, and only 15% credited the agent.
Anthropic published a study on October 27 about how much experts really decide when they work with an AI agent. In a randomized trial, 50 professional analysts chose the same option as the lean the researchers had secretly assigned to their agent in 86% of the analytical decisions, against the roughly 50% that would be expected if the agent had no influence, according to the study's official page.
Anthropic recruited 61 professionals with experience in causal inference, that is, in using data to judge whether one factor drives another; 11 opted against using the agent and were excluded from the main analysis. Each participant advised a fictional farming cooperative on whether to expand a voluntary, paid advisory service. They worked with Claude Code, Anthropic's AI agent for coding, for a median of 65 minutes, and made four analytical decisions with two reasonable options each. The agent's preference on each decision was set at random and given to it through hidden instructions; the researchers deliberately strengthened those preferences.
The match rate was 92%, 82%, 81% and 90% on the four decisions. Where an analyst had stated a preference of their own beforehand, they abandoned it in 60% of cases when the agent disagreed and in 6% when it agreed. Later, unable to reread their chats with the agent, 77% reused its arguments in at least one answer, at times verbatim, yet just 15% named the agent as their source. Anthropic concludes that keeping a person in charge of a process assumes that their judgment is an independent check, and that this holds only if the judgment is their own.
The study has limits that the company itself acknowledges: a single task, in one field and with one model; the agents' preferences were deliberately strengthened, so without that the effect might be smaller; and the figures are averages across analysts, not a measure of how much the agent influenced each person.
Why it matters · analysis and opinion
An expert's sign-off does not guarantee that the expert made the decision. The study suggests that, when an agent argues for a preference with confidence, the person tends to adopt it and later to explain it as their own, so human review loses value as an independent check. What is still unknown is how much this weighs in real work: the task was a single one, with one model, and the preferences were strengthened on purpose, so the everyday effect could be smaller. For a small business or a professional who already works with agents, a reasonable rule, as a matter of judgment, is to write down one's opinion before reading the agent's and to decide deliberately which decisions should stay in a person's hands.
Official source: Anthropic · Written with the help of AI: how we make the news
