Study: People Who Get AI Advice Are Less Accurate — But More Confident

"People became much worse. The accuracy was only one third, but they were twice as confident." That's how Valerio Capraro, one of the authors of a new study on AI-assisted decision-making, summed up what happened when his research team gave people access to an AI chatbot before asking them questions the chatbot was likely to get wrong.

How the study was built

Capraro, working with Chiara Marcoccia (École Normale Supérieure) and Walter Quattrociocchi (Sapienza University of Rome), ran five experiments with 3,132 participants in total, four preregistered plus one direct replication, according to a paper posted to arXiv. To isolate the effect of simply having AI available, as opposed to AI actually being helpful, the researchers deliberately picked questions an AI chatbot tends to answer incorrectly: fine-grained visual details from films, such as the color of a team's kit in a specific scene of "Bend It Like Beckham."

The numbers

Without AI access, participants answered correctly 27% of the time and reported 30% confidence on average, a reasonably honest match between how right people were and how sure they felt. With AI access, accuracy dropped to 9%, roughly a third of the baseline, while confidence jumped to 76%. Perhaps the most striking shift was in participants' willingness to simply say "I don't know": in the no-AI condition, 44% of participants admitted they didn't know an answer rather than guess. With AI available, that fell to 3%.

Adding a financial incentive for accuracy helped, but only partially. Paying people for correct answers raised "I don't know" responses from 3% to 8%, and accuracy from 9% to 16%, both still well below the levels people managed with no AI involved at all.

Why this matters beyond trivia questions

The researchers frame this as evidence that AI availability doesn't just occasionally mislead people, it actively suppresses the instinct to recognize the limits of one's own knowledge. That instinct, choosing to say "I'm not sure" instead of guessing, is exactly what's supposed to prevent bad decisions from compounding in professional settings: a doctor, an analyst, a customer support agent.

Reading between the lines

The uncomfortable implication for any business rolling out AI assistants isn't that the tools are unreliable in some narrow technical sense; it's that people using them may become systematically worse at knowing when to double-check. Practical mitigation is less about picking a "smarter" model and more about workflow design: build in friction, a mandatory second source, a visible confidence flag, a rule that certain categories of answers get verified regardless of how confident the tool sounds, rather than trusting employees to self-regulate against a tool that's engineered to sound assured either way.

Sources: TheNextWeb