Nik Martelaro
Associate professor at Carnegie Mellon University's Human-Computer Interaction Institute, researching how AI tools can support designers' thinking and creative...
“86% of the time, analysts went with whatever the AI favored. Now, if the AI didn't have an influence on this, you would actually expect that they would go 50-50. … And they often would, when asked why, give the AI's reasoning. They wouldn't even reference the AI's reasoning. They would just say, this is why I believe this. But actually, it was exactly what the AI had said.”
AI and Design · AI Builders · November 2026
Context
Martelaro is summarizing an Anthropic study in which experienced data analysts used Claude Code on a business question, and researchers randomly chose which reasonable option the AI would argue for. Analysts followed the coin-flipped recommendation most of the time and repeated its reasoning as their own.
Transcript
…of the situation. And so, I mean, it's cool to see though. It seems like the paper does suggest that though they were able to overcome some of those challenges and that it did work. So who knows? We might be getting there. But I do, I do worry about that. I worry about how brittle it can become.
…with the transformer models and all the other new models that have come out subsequently, it's broken through that barrier. And now you can have these more contextual flowing things. The if-then statements, the massive logic trees are interesting because they're so readable, but they are limiting.
So as we talk a little bit about the fact that AI also can, when it understands something about us, could potentially intervene or steer us. This actually leads us into our last article of the day in a new report from Anthropic, which suggests that people, even experts, might be leaning a bit more on the decisions that AI makes than they even think they are doing. So in this report, Anthropic gave 50 experienced data analysts a realistic job using cloud code Opus 4.8. And basically these people have skills in statistical analysis and in looking at data to then develop systems to help them make decisions. And what they did was they made up a farming co-op that wanted to know whether its advice program for farmers was working and whether to expand it. And along the way, what happened was the analysts have to make four judgment calls where either answer was reasonable. So it wasn't like, oh, they put in like, okay, one option is a good option. One option is a terrible option. Okay. Can we now figure out how to get the experts to make the bad option? It's more realistic, right? Oftentimes the data suggests to you that there could be multiple directions to go…
And when the AI disagreed with them, they changed their minds more frequently more than half the time.
Right. Now, one of the interesting things that they did in this was sometimes they would ask the analysts to think about it and make a decision ahead of time. And then they would present the AI's position. And in that case, people only went with the AI's answer 60% of the time. So thinking ahead…
This reminded me a lot of our talk with Steve Rathjay a couple weeks ago on the podcast where he was talking about AI sycophancy and how humans are easily convinced that this thing is correct, especially when it's sycophantic. And they do say that in this test, they did make the AI slightly pushier…
More from AI and Design
