“AI is really good at surfacing themes, but less good at that nuanced context. … Sometimes people contradict themselves, but you have to be looking at all of the little details to understand what was the final thing they were coming away with. … There's maybe some outliers that are important that get thrown away. Or it's too homogeneous. The results are just flattened a little bit. And then of course there are hallucinations. They're still making stuff up. We see that all the time still.”
sputnikintelligence.com
Asked whether faster AI-driven research risks information overload without the judgment to match, the bank UX researcher lists what machine-synthesized research still misses, and why human researchers are still needed.
…a clear answer. And it's also if there's like a verbatim clip that you can pull from your user testing research that like really crystallizes a person's struggle with a thing or the thing that they like, and you hear another person talking about it, like, I think it really clicks for stakeholders.
…have grown because the technology enables us to scale up what it is we do. And so I guess, do you think we're at risk of getting too much information, just being inundated with a lot of information, but not necessarily having the complementary judgment to be able to focus on what matters?
Yeah, those are really salient points. So what we're witnessing now with the growth of these tools that we all have now access to is that there's a now sort of roles are becoming a little more fuzzy, right? Like as a designer, I can also start building HTML prototypes. I can start doing, you know, like numbers analysis that I wouldn't typically be doing, right? And developers can start designing screens. And, you know, everybody has these tools to do the data analysis for them, right? So when you look at the outcomes, though, right? If I get a design prototype from a developer, sometimes it's really good, but sometimes at first glance, it's like very impressive. But then when you start like digging into the details, you're like, oh, that doesn't make any sense, or that's kind of a miss. Or, right. So the devil's always in the details. And you realize that that subject matter expertise is needed, right? To do the designs. Or, you know, so what I'm getting to is about the sort of like the rigor in the analysis of the research, right? So I think so far from what I've seen, AI is really good at sort of surfacing themes, but less good at that sort of nuanced context that, you know, a transcript that's processed by a machine can really pick up. So I think that there's a few…
I think for good reasons that humans will still be a part of that process. Yes. I think. But actually, kind of building on that point, as we are seeing AI being integrated into the tools that we're using to execute on design, how do you help to distinguish between some of these suggestions that…
I mean, I think that the biggest example of this is people using chatbots to surface like a poor information architecture, really, right? Like some information that you can't easily find. So you just use the chat agent to get right to it. And maybe people would choose to do that even if you did…
Yeah. And, you know, it's interesting because there's just a lot of conversation around service design and really thinking through how we coordinate these multi-channel experiences, right? Yeah. We've got voice and screens and, you know, to a greater extent, even like people in physical locations…