Six key takeaways from a masked AI vs human test
Ogilvy and Lloyds Banking Group recently staged an
‘AI Turing Test’ to literally answer the question:
can AI truly replace human creativity?
Six key takeaways from a masked AI vs human test
Ogilvy and Lloyds Banking Group recently staged an ‘AI Turing Test’ to literally answer the question: can AI truly replace human creativity?
The idea was born over lunch. Suresh Balaji, chief marketing officer at Lloyds Banking Group, was dining with David Golding, executive planning partner at Ogilvy Group UK, when a waiter informed the table the restaurant had no Diet Coke but could offer Pepsi Max instead. Balaji wondered aloud whether he would actually be able to tell the difference.
That sparked a conversation about masked tests which, given the company at the table, quickly escalated into a bigger question:
"What if we tested AI against human creativity - and asked if people could tell the difference?"
Talk became action, and in November Ogilvy and Lloyds conducted a bold experiment: an AI Turing Test applied to real-world marketing. Inspired by Alan Turing’s famous challenge to distinguish human from machine, Ogilvy was tasked with developing a full-funnel campaign for Lloyds’ new travel-booking service - including strategy, messaging and creative.
“What made this test unusual,” Golding joked, “was that we did it properly. Historically, in ad land, we’d just lash something together. But everything was done scientifically - or as scientifically as you can in an agency."
For Balaji, the goal was simple: discover whether AI is genuinely taking creative jobs away, or whether it can augment human creativity. “This test gave us a proper answer,” he concluded.
The experiment setup
Three independent teams worked in parallel. A human-only traditional creative team with no AI assistance. An AI-powered human team augmented with AI tools, using AI to explore ideas, generate options and test messaging. And a fully AI-driven team and workflow, operated by prompt engineers through WPP’s AI tool WPP Open, with minimal human creative input.
All teams received identical background materials - brand guidelines, product details and audience insights - and worked in isolation. To keep things rigorous, Golding said the agency “had perfectly run meetings, a panel of judges and filmed everything.”
The results
At the IPA event where the work was unveiled, the audience was first shown the three strategic routes and asked to vote on which one they believed was fully AI-generated.
The votes were split, with people unsure as to which route contained the AI copy - a telling sign of AI’s developing strategic capability.
But when the creative visuals were revealed, the distinction became much clearer. The audience could generally identify which executions came from humans, which were generated by AI, and which blended the two.
Six key takeaways
1. AI-powered human
creativity hits the sweet spot
The combination of human intuition and AI capability outperformed the other two approaches overall. The hybrid team maintained momentum, rapidly tested multiple concepts and developed an end product without losing sight of brand values or emotional impact.
"The human and AI combo provides the best results, combining scale and creativity,” Ed Turner, head of strategy at Ogilvy One, explained. “AI alone can produce content, but human guidance is essential to bring ideas to life."
Chris Davis-Coward, head of consumer marketing at Lloyds, highlighted the importance of human oversight, noting that AI was brilliant for production and scaling ideas, but that it struggled to generate fully polished creative immediately. “We had to actively guide it and assess it against each brief - a challenge in collective leadership."
“We had to actively guide it and assess it against each brief - a challenge in collective leadership."
Chris Davis-Coward,
Head of Consumer Marketing, Lloyds
2. AI’s strategic and insight-
generation ability is impressive
"AI highlighted that the average number of tabs someone has open when booking a holiday is twelve, That insight informed creative ideas and product development, showing behaviours that might not have been evident otherwise."
Ed Turner,
Head of Strategy, Ogilvy One
AI demonstrated an exceptional ability to analyse data and connect seemingly unrelated insights to generate strategic thought that felt surprisingly human. The IPA audience’s difficulty in spotting the AI-generated strategy reflected this.
More specifically, a standout example came from the AI-only team, which surfaced a behavioral insight that might have been missed. "AI highlighted that the average number of tabs someone has open when booking a holiday is twelve,” Turner added. “That insight informed creative ideas and product development, showing behaviors that might not have been evident otherwise."
3. Human oversight and
creativity remains crucial
For all of AI’s strengths in insight and scale, it faltered in areas requiring nuance, humour, emotional resonance and brand alignment. Eva Steiner, senior art director at Ogilvy One, admitted she initially feared the test might argue for her replacement. “But quickly, I realised I was going to be just fine,” she added.
The AI-only team’s output fell apart when producing imagery and it ultimately struggled to shape ideas with emotional subtlety. At one stage, a meeting had to be delayed as the AI agents were taking longer than expected.
"AI will eventually produce outputs indistinguishable from human work,” Johnny Watters, executive creative director at Ogilvy One, affirmed. “But the truly standout creative ideas - the ones that differentiate brands - will always need humans."
Ogilvy and Lloyds could see what the AI team was attempting to achieve but it still required a human nudge. Davis-Coward emphasised: “Its potential is significant, but it’s not fully autonomous in creativity."
"AI will eventually produce outputs indistinguishable from human work but the truly standout creative ideas - the ones that differentiate brands - will always need humans"
Johnny Watters,
Executive Creative Director, Ogilvy On
4. AI enhances team
collaboration and confidence
“It enables a little more positive provocation in a room where people might not normally speak up."
Ed Turner,
Head of Strategy, Ogilvy One
Perhaps the most unexpected outcome was how AI improved team dynamics. In the hybrid team, AI acted as a neutral mediator - offering immediate feedback, challenging assumptions and helping resolve debates without ego.
Turner described this phenomenon as “agentic confidence”. He added: "Using AI as a mediator allows teams to disagree agreeably. You can test ideas quickly and get constructive feedback without conflict. Teams can rally around an idea with more conviction."
For Steiner, AI encouraged collaboration, giving a fresh perspective without creating friction. “It enables a little more positive provocation in a room where people might not normally speak up."
The result was more exploration, more experimentation and a more confident creative process.
5. Embracing novelty
improves engagement
According to the hybrid team, introducing AI into the creative process made the work more playful and energising. The novelty factor encouraged experimentation and helped the teams explore concepts they might have otherwise dismissed. It also fostered a sense of excitement and curiosity, reminding participants that creative work can be both productive and enjoyable when new tools are embraced.
"The human and AI collaboration was a shift in how we work together,” Chris Davis-Coward added. “It brought fun, play and creativity."
"The human and AI collaboration was a shift in how we work together,”
Chris Davis-Coward,
Head of Consumer Marketing, Lloyds
6. What this means
for the industry
The AI Turing Test provided a mostly clear industry lesson: AI works best as an enhancer of human creativity, not a substitute. It can accelerate production, surface insights and build team confidence but the ideas that truly cut through still depend on human intuition, judgment and emotional understanding.
“AI alone doesn’t guarantee speed; human involvement is often required to get from concept to execution,” Turner said. “The sweet spot is where humans and AI collaborate.”
This experiment also points toward a new agency model. Watters noted that AI agents are becoming increasingly prevalent across the creative process. “They give immediate confidence, but humans provide the top layer that delivers originality and emotional resonance.”
For Lloyds, the test redefined AI’s place. “We’ve reframed AI as part of the team,” Davis-Coward explained. “Not just a tool for speed and scale, but a collaborator.”
For the wider industry, the message is simple: the future is hybrid - AI accelerating the work, and humans ensuring it means something.
Built with Shorthand