The rise in AI-powered cyber attacks signals a shift in the cyber threat landscape, where traditional security measures often struggle to keep pace. Individuals, enterprises, critical infrastructures, and governments alike now face risks that are amplified by the intelligent capabilities of AI-driven tools. Understanding how AI is weaponized is no longer optional; it is essential for building resilience in an increasingly digital world. The Global Cybersecurity Outlook 2025 Report by World Economic Forum highlights the "AI–cyber paradox": while 66% of surveyed organizations believe AI will have the greatest impact on cybersecurity in the next year, only 37% currently have processes to evaluate the security of AI tools before deployment. Additionally, it highlights the risks associated with the use of AI as an offensive tool, noting that Generative AI is augmenting cybercriminal capabilities, contributing to an uptick in social engineering attacks.

Cybercriminals are weaponizing AI to automate and amplify attacks, using it for highly convincing phishing campaigns, deepfake scams, automated malware generation, and large-scale data theft, etc.

This report delves into different case studies that illustrate the evolving use of AI in cyber attacks, offering insights into the emerging challenges and the urgent need for innovative detection and defense strategies.

Case Studies of AI-based Cyber Attacks

• AI-driven social engineering attacks

Social engineering has long been a powerful tactic for cybercriminals, but with the rise of AI, these attacks have become faster, more advanced, and harder to detect. The attacks now encompass a spectrum of tactics, including highly personalized phishing emails, synthetic media generation such as deepfakes and voice cloning, and sophisticated impersonation schemes. Attackers use AI to automate reconnaissance, craft convincing messages, and dynamically adapt their strategies in real time, making detection and prevention increasingly challenging.

Phishing: AI has significantly transformed phishing attacks, making them more sophisticated, personalized, and harder to detect. Cybercriminals now leverage AI to automate and enhance their phishing campaigns, posing a growing threat to individuals and organizations alike.

A survey of security leaders in the U.S. and U.K. revealed that 93% expect daily AI-driven cyberattacks by 2025, with 38% specifically anticipating attacks involving AI-powered phishing. As per Perception Point’s ‘2024 Annual Report: Cybersecurity Trends & Insights’, there has been a 1,760% year-on-year increase in social engineering-based Business Email Compromise (BEC) attacks, highlighting the growing reliance on AI tools by threat actors (Reference). According to research by SoSafe, Generative AI tools speed up phishing attacks by at least 40%. Another study by IBM researchers on way back to 2023 also mentioned the use of ChatGPT in creating highly convincing phishing emails in minutes. In a test involving 1,600 employees of a global healthcare company, 14% fell for a phishing email crafted by ChatGPT.

Keeping in view the recent advancements in AI, this presents significant challenges for cybersecurity professionals to enable strong defences against such attacks.

Synthetic Media and Deepfakes: The emergence of AI has led to increased capabilities in synthetic media generation. Particularly deepfakes, which include AI-generated audio, video, or images that mimic real individuals, have become potent instruments in cyber attacks. These technologies facilitate advanced social engineering tactics, enabling attackers to deceive, manipulate, and exploit targets with unprecedented realism. The research mentions several case studies involving use of synthetic media for cyber attacks. In 2019, one of the earliest documented cases involved a whale-phishing campaign that leveraged synthetic media techniques. Cybercriminals used AI-generated voice technology to impersonate a CEO’s voice, successfully instructing a subordinate to transfer $243,000 to a fraudulent account. This form of voice phishing, or “vishing,” exemplifies how synthetic audio can be weaponized to execute high-stakes financial fraud by mimicking authoritative figures (Reference). In 2024, a multinational finance company lost $25 million after attackers used real-time deepfake video conferencing to mimic executives (Reference). The deepfake combined synthetic audio and video to approve fraudulent transactions, bypassing human and technical safeguards. This case underscored the feasibility of live deepfake manipulation in high-stakes environments.

Further, there have been reports about AI-generated ID documents, bypassing well-known KYC softwares. A service named OnlyFake leveraged AI to create counterfeit passports and driver’s licenses for 26 countries, priced as low as $15. These IDs successfully bypassed Know Your Customer (KYC) checks on crypto exchanges like OKX, Kraken, and Revolut, enabling anonymous account creation and money laundering. Recently, a Polish researcher used ChatGPT-4o to create a highly realistic fake passport (Reference). The AI-generated document bypassed automated KYC checks on major platforms like Revolut and Binance, raising concerns over identity verification, legal breaches, and potential misuse of personal data.

Hacker groups are also adopting synthetic media to infiltrate tech firms. These hackers use generative AI in their malicious interview campaign - a series of tactics that involve gaining employment in remote technical roles in western firms, usually in industries with sensitive security data like defense, aerospace, or engineering (Reference). As an example, a recent case reports the use of a Polish politician's face by North Korean Hackers to develop fake profiles.

These incidents collectively highlight the escalating threat posed by synthetic media in cyber operations. The realism achieved through deepfakes can erode trust in digital communication and challenge traditional security infrastructures. Proactive defense strategies and broad-based awareness are essential to mitigating the rising tide of such deepfake-enabled cyber threats.

The convergence of AI and social engineering is not limited to email scams or deepfakes; it extends beyond initial access and deception. Increasingly, attackers combine these methods with malware deployment, using social engineering as the entry point for ransomware and data exfiltration. For example, in September 2023, the hacker group ‘Scattered Spider’ exploited social engineering tactics to breach MGM Resorts. By impersonating an employee via phone, aided by AI-generated personas, they tricked the help desk into resetting multi-factor authentication credentials. This allowed the attackers to gain access to high-value systems, deploy malware, and cause service disruptions costing the company huge damages (Reference). Such cases illustrate how AI-enhanced social engineering is often just the entry point to larger attack chains involving ransomware, lateral movement, and data exfiltration. As these technologies become more accessible, organizations must move beyond traditional security awareness to deploy AI-driven defense systems, integrate behavioral anomaly detection, and enforce strict identity verification protocols across digital channels.

• Malware Development

The emergence of AI-powered tools has fundamentally changed the landscape of malware development. Once reliant on static code and human scripting, modern malware is now dynamic, intelligent, and adaptable. Threat actors are increasingly deploying AI to generate malicious code, automate exploitation, and evade detection with unprecedented precision and scale.

Dark GPTs: A growing threat comes from AI language models trained and repurposed specifically for cybercrime. Tools such as WormGPT, FraudGPT, WolfGPT, and XXXGPT have emerged on dark web marketplaces and Telegram channels, providing cybercriminals with the ability to write polymorphic code that evolves to bypass security tools; generate phishing emails and malware scripts tailored to specific targets; craft exploits for known vulnerabilities; assist in scam operations, such as business email compromise (BEC). The underground market for these tools is thriving, with threat actors like "CanadianKingpin12" identified as prolific developers and sellers of dark LLMs across crime forums (Reference).

Polymorphic and Adaptive Malwares: AI has enabled the rise of polymorphic malware, which is a malicious code that constantly evolves its structure to avoid detection. An example of this advancement is BlackMamba, a proof-of-concept keylogger that uses a large language model (LLM) to dynamically generate its malicious functionality at runtime. It requires no command-and-control infrastructure, instead synthesizing polymorphic payloads in memory. This dynamic approach makes traditional signature-based detection methods increasingly ineffective. Adaptive malware goes a step further by altering its execution patterns and communication protocols in real time based on the security environment it encounters. This makes it more persistent and capable of surviving sophisticated defense mechanisms.

Malicious AI Models in the Wild: The proliferation of malicious AI and machine learning models is not limited to underground forums. In 2024, researchers discovered hundreds of compromised AI/ML models on legitimate platforms like Hugging Face, highlighting the risks of supply chain attacks and the potential for widespread distribution of backdoored or weaponized models (Reference). As of April 2025, Protect AI and Hugging Face reported scanning 4.47 million unique model versions and identifying 352,000 unsafe or suspicious issues across 51,700 models on the Hugging Face Hub.

AI-driven malware development represents a paradigm shift in cyber threats, combining speed, adaptability, and scale. As these tools become more accessible, organizations face a growing challenge in defending against rapidly evolving, AI-powered attacks that can bypass traditional security measures and inflict significant damage.

• Vulnerability Expansion

AI has also revolutionized the field of cybersecurity, offering powerful tools for identifying and mitigating vulnerabilities within complex systems., enabling defenders to detect, classify, and prioritize security weaknesses with unprecedented speed and accuracy. However, the same technologies that bolster security can be repurposed for malicious intent.

AI-Induced Expansion of the Attack Surface: The integration of AI into various systems has inadvertently expanded the cyberattack surface. Agentic AI systems, which operate autonomously and make decisions without human intervention, introduce new vulnerabilities. These systems can initiate complex chains of interactions, some of which may be exploited by adversaries. A recent study emphasized that the non-deterministic nature of such AI models complicates security efforts, as their unpredictable behaviors can be manipulated to bypass traditional defenses.

Slopsquatting: This represents a novel AI-driven supply chain attack vector where threat actors exploit generative AI's tendency to hallucinate non-existent software packages during code generation. When developers, relying on AI tools like ChatGPT or GitHub Copilot, attempt to install these fictitious packages, they may inadvertently introduce malicious code into their systems. A comprehensive study analyzing over 576,000 code samples from various AI models revealed that approximately 20% of the suggested packages did not exist. Notably, open-source models exhibited a higher hallucination rate (21.7%) compared to commercial ones (5.2%). This predictability allows attackers to preemptively register these non-existent packages with malicious payloads, anticipating that developers might attempt to install them.

Image Source: arxiv.org

As an example, a researcher registered a fictitious package named ‘huggingface-cli’, which was suggested by AI tools instead of the legitimate ‘huggingface_hub’, for which the correct installation command should have been: pip install -U "huggingface_hub[cli]". This deceptive package was inadvertently included in the README of Alibaba's open-source package, GraphTranslator, highlighting how easily such vulnerabilities can propagate.

Adversarial AI: Adversarial AI exploits vulnerabilities in machine learning (ML) systems by manipulating inputs, models, or outputs to cause misclassification, data breaches, or system failures. These attacks target the core logic of AI systems, often bypassing traditional cybersecurity defenses. Adversarial examples have been demonstrated across various domains, including image recognition, speech processing, and reinforcement learning, with attackers using different techniques to bypass security controls. The risks posed by adversarial AI are significant. For instance, attackers might slightly alter an image so that a self-driving car’s AI sees a stop sign as a speed limit sign, which can lead to dangerous situations. Similarly, adversarial attacks on speech recognition systems can induce misinterpretation of voice commands, and reinforcement learning agents can be manipulated to make suboptimal decisions through carefully crafted environmental changes.

Thus, the use of AI, especially with the evolution of LLMs, has amplified the cyber risks. The research further details the underground exploitation of LLMs for malicious services (defined as the term, “Malla” for such activities).

Image Source

As artificial intelligence continues to transform the cyber threat landscape, it is imperative that defense mechanisms evolve in tandem with these advancements. Organizations and individuals alike must recognize that AI-powered attacks, from deepfakes and social engineering to adaptive malware and adversarial manipulation, demand a new level of vigilance and preparedness. Robust cyber hygiene practices, such as regular software updates, strong authentication protocols, and careful handling of confidential data, are more critical than ever. At the same time, leveraging AI-driven security solutions is essential; as AI can only match the speed, scale, and sophistication of AI-enabled adversaries.

Ultimately, building cyber resilience in the age of artificial intelligence requires a proactive, adaptive approach that anticipates evolving attack vectors and integrates advanced technologies to protect digital assets and maintain trust in our interconnected world.

Acknowledgements

This report has been prepared as part of project assignment for the AI Safety, Ethics, and Society Course Spring 2025 offered by Center for AI Safety. Special thanks to the course team, Anders Edson, course facilitator Judita Ru, and cohort team members including Indra Gesink, Alde González, Shun Yoshizawa, Bach Nguyen, and, Paweł K for their continuous support.

This work is the result of independent research conducted from multiple sources. AI tools including ChatGPT and Perplexity have also been selectively used at certain points only to support linguistic phrasing and clarity, without influencing the core content or analysis.

Subscribe now