The boom in generative Artificial Intelligence tools in recent years has been transformative.

But the speed of AI’s adoption by the public has meant education in them has struggled to keep up. Now, warnings are being issued about the potential dangers to our money, health and security of using AI tools without due caution.

Here’s what you need to know. 

While recommendations and numbers you receive from the Bank directly are safe and compliant, those you receive from other sources might not always be.

Still, an increasing number of us use Artificial Intelligence platforms such as ChatGPT, Claude or Gemini, as if they were search engines – despite clear and repeated warnings, including from the platforms themselves, that AI chatbots hallucinate answers, and cannot be relied upon to give accurate statements, however plausible they sound or authoritative the language they use.

This tendency to trust AI chatbots for real-life, practical queries is leading to trouble, as scammers are stuffing AI results with deliberately misleading information that can take you to false customer service numbers or online services that impersonate the organisation you’re looking for. 

They use a technique called ‘poisoning’, in which malicious actors seek to subvert the output of generative AI by corrupting its commands or feeding it bad information in advance of your query.

So how does it work? 

This practice is increasingly used by bad-faith actors from the world of statecraft and propaganda, organised crime, and even fringe politics, to feed people who use AI to search, deliberately wrong answers.

These might give the scammers’ own contact details instead of those of the business or organisation the user is looking to contact.

Equally, they might give false product information, fake feedback (good or bad) to show a competitor’s service or their own in whatever light favours them.

This means the contact details you are searching might belong to phone lines or deceptive websites or social channels set up by the scammers themselves. 

So-called Generative AI tools are not truly intelligent, but are based on something called a Large Language Model (LLM). Put simply, a LLM is fed huge volumes of language, interactions and information from around the web, and will serve back to you what it calculates is the most likely combination of those words, phrases, pixels or other pieces of information to satisfy your prompt. This it frames as the answer to your query.

This is different to typical search engines.

Google (for example) treats credibility and trustworthiness as a weighting when deciding which website to offer as a solution to your query, along with factors like sheer volume of mentions, load speed and domain. What’s more, they ‘downrank’ misinformative content, and penalise impersonator sites. The system is not perfect. But by offering the chance to look at the sites in question, their addresses, and their metadata, and by ranking how much it trusts them, it does offer you a host of information on which to weight up their trustworthiness.

In contrast, AI tools mostly work with ‘Zero-Click Impressions’. That is, they summarise information on web pages, without directing you to those pages themselves. Additionally, they tend to ‘hallucinate’ answers, giving false information, even if the information they are ingesting is accurate. For instance, claiming that there should be only two ‘r’s in the word ‘strawberry’.

That makes them unreliable for searching out accurate, trustworthy answers to queries of fact.

It also means that scammers can bombard the term – for example, ‘What is the best number to call for a financial service in the UK?’ – with web traffic giving false answers.

They do this, knowing that there is a chance that each time the query is asked, the AI engine will scrape the internet for what look like answers, and offer them up – without checking if they are true or not. 

Here is one recent, real-life encounter with poisoned AI results (edited for anonymity and style).

“I wanted a refund from a company, and asked my AI the best way to get it. It answered and gave me the phone number with a script. I called it, thinking it was the company. I gave my name and address, and described the situation. They told me they would give me credit that could be used at retailers.”

The caller had in fact been put through to the scammers’ own phone line, given as the answer by the AI tool. The victim only became suspicious when the purported call-centre representatives began asking unusual details. “They started asking for my routing number, which is when I got suspicious. They couldn’t answer why, so I hung up.”

The scammers had been able to get their number given as the ‘right’ one for the legitimate company by pumping the web full of low-quality content with the fake number on, knowing that many AI bots would offer them up as the most frequent and therefore ‘most likely to be true’ answer.

This user did what they should have done in the first instance, and checked the company’s own channels to source the correct number. 

The bottom line is, there are many benefits to AI tools; but be mindful of using as a trusted source for contact details, essential information, reliable facts, contact details or financial actions of any kind.

This is a message worth sharing. Use of AI tools continues to rise. Research this spring by Saga revealed that 30% of Britain’s over-50s would consider using AI tools to plan holidays. Recent news has shown people trusting AI answers for everything from contact numbers to financial, medical, and legal advice, sometimes with negative results.

These tools may offer inspiration, but always double-check any ‘hard’ information they offer. And of course for any contacts at the Bank, your Premier team can help refer you onwards.

Remember, always try to use the numbers that come directly from the organisation – and if in doubt, just give us a call.