Explainer · Adapt Better
How AI raises the social engineering risk
AI does not invent new human weaknesses. It industrialises the old ones, making the same manipulation cheaper, faster and far more convincing.
The short answer
In plain terms
AI has not changed the levers attackers pull. Urgency, authority and helpfulness are still the target. What it has changed is the cost and the polish: convincing fakes at scale, with the tells that used to trigger doubt removed.
What you may be seeing
How it shows up
A voice or face on a call that looks exactly like a colleague. Phishing with no clumsy grammar to give it away. Chatbots and assistants talked into ignoring their own rules. And the AI tools themselves becoming something worth attacking.
Underneath
What is really happening
Social engineering works by loading the moment, pressure, a senior name, a reason to help, so the shortcut runs before judgement catches up. AI removes the friction that used to interrupt that: the wrong face, the odd phrasing, the delay. It turns out a model has its own version of the same weakness too, its guardrails can be talked around much as a person’s caution can. And the AI supply chain, the models and tools teams now trust, is a fresh surface to poison.
What helps
What actually moves it
The defence changes less than it looks. The behavioural move still holds: slow the urgent request, verify on a second channel, make it normal to check. What is added is treating AI as attack surface too, provenance for the tools and models you adopt, and basic secrets hygiene, so a convincing fake meets a habit that does not bend under pressure.
The evidence
These are recent, documented cases, cited to show the pattern rather than to single anyone out; every platform in this space faces the same pressure. In February 2024 a finance worker at the engineering firm Arup was deceived into transferring about US$25 million after a video call in which every other participant was an AI-generated deepfake of senior colleagues. In August 2024 researchers showed Meta’s open-source prompt-injection filter, PromptGuard, could be walked past simply by spacing out the letters of a malicious instruction, dropping its detection rate from near 100% to under 1%; Meta acknowledged the issue. And in 2024 the AI model hub Hugging Face disclosed unauthorised access that may have exposed a subset of stored secrets, while security firms separately found models uploaded to the hub that run hidden code when loaded.
- CNN, 4 February 2024. Finance worker pays out $25 million after video call with deepfake ‘chief financial officer’. www.cnn.com/2024/02/04/asia/deepfake-cfo-scam-hong-kong-intl-hnk
- SC Media, 2024. Meta’s PromptGuard model bypassed by simple jailbreak, researchers say. www.scworld.com/news/metas-promptguard-model-bypassed-by-simple-jailbreak-researchers-say
- The Hacker News, June 2024. AI Company Hugging Face Detects Unauthorized Access to Its Spaces Platform. thehackernews.com/2024/06/ai-company-hugging-face-notifies-users.html
Where this leads
The programme behind this is Social Engineering: Think like a hacker. For anything specific to your setting, the fastest route is a conversation.
Filed under Adapt Better. By MindHug. Last updated 2026-09-24.