Social engineering attacks are one of the most commonly used methods for cyber criminals to gain access to sensitive data and information. Recent advances in generative AI have enabled threat actors to utilize AI-generated content to fuel these attacks, making them more convincing and harder to detect. In this blog post, we will discuss how threat actors are leveraging generative AI to facilitate social engineering attacks and credential compromise.
What is Generative AI?
Generative AI is a subset of artificial intelligence that utilizes machine learning algorithms to generate new data that closely mimics existing data. This technology has the potential to transform numerous industries, including cybersecurity. Generative AI can be exploited by threat actors for social engineering and initial access, which is why understanding this technology is crucial for businesses looking to protect themselves from cyber threats.
In the realm of cybersecurity, generative AI can be used for both defensive and offensive purposes. On the defensive side, it can be employed to detect and prevent cyber threats by generating synthetic data to train AI models and test for vulnerabilities. However, threat actors have also recognized the potential of generative AI in carrying out social engineering attacks. Both threat actors and cyber defenders use generative adversarial networks (GANs), which involve two components: a generator and a discriminator. The generator is trained on a dataset and learns to generate new data that is indistinguishable from the original. The discriminator, on the other hand, tries to differentiate between the generated data and real data. Through an iterative process, the generator and discriminator continuously improve, creating more and more realistic outputs.
The Intersection of Generative AI and Social Engineering
Social engineering attacks involve manipulating individuals or groups to divulge sensitive information or perform actions that can lead to a security breach. By leveraging generative AI, threat actors can create highly realistic personas, messages, and scenarios to deceive their targets. For example, they can generate fake social media profiles that closely resemble real users, send convincing phishing emails, or create realistic voice recordings for phone-based scams. Threat actors have created malicious AI tools such as WormGPT and FraudGPT for these purposes.
The main challenge with generative AI-powered social engineering attacks is that they can bypass traditional security measures and fool even vigilant users. As technology improves, it becomes increasingly difficult to distinguish between real and fake content, putting individuals and organizations at risk. People are generally more vulnerable to manipulation than machines, and threat actors can take advantage of this vulnerability to steal data or gain access to systems.
By training AI models to mimic human behavior and language, threat actors can create highly convincing fake personas that can be used to manipulate their targets. For example, a threat actor could create a fake LinkedIn profile that appears to be a real person, complete with a photo, work history, and connections. This profile could then be used to initiate conversations with targets and build trust over time, leading to a successful social engineering attack.
Another way that threat actors can use generative AI in social engineering attacks is by creating highly convincing phishing emails. By using natural language processing and other techniques, they can create emails that appear to be from trusted sources, such as banks or other financial institutions. These emails can be personalized to include specific information about the target, making them even more convincing.
Overall, the intersection of generative AI and social engineering is a concerning development in cybersecurity. As threat actors become more sophisticated in their use of these technologies, it is critical for organizations and individuals to increase their cyber awareness and take steps to mitigate the risks.
Generative AI and Credential Compromise
One of the most common results of generative AI-powered social engineering attacks is credential compromise. Credential compromise happens when a malicious actor obtains usernames and passwords, typically for the purpose of gaining access to their accounts or sensitive information. Threat actors employ various techniques, such as credential harvesting and credential stuffing, in order to compromise accounts.
Credential harvesting involves using social engineering in order to trick users into entering their credentials into a fraudulent login page, at which point they are acquired by the attacker. Generative AI allows threat actors to create fake login screens or mimic legitimate websites, fooling individuals into entering their credentials. These fake screens can be virtually indistinguishable from the real ones, making it difficult for users to detect the deception.
Credential stuffing, on the other hand, involves using stolen or leaked login credentials from one website or service to gain unauthorized access to another. Threat actors use generative AI to generate the code to automate or create malware, greatly increasing the speed and efficiency of their attacks.
Once threat actors have successfully compromised someone’s credentials, they can gain access to sensitive information, financial accounts, or even take control of entire systems. This can have severe consequences for individuals and organizations, including financial loss, identity theft, and data breaches.
Protecting the Human Attack Surface to Combat this Threat
Not everything means doom and gloom for cyber defenders in the new age of generative AI. Threat actors use open source intelligence (OSINT) from the open, deep, and dark web to conduct reconnaissance, plan social engineering and credential compromise attacks, and build the infrastructure to develop resources to deceive, scam, or coerce employees into performing actions that put their employers at risk of a breach. To fulfill their tasks, AI tools require what Picnic calls “target intelligence”, which means all the publicly available information on their targets: employees and connected infrastructure. The only solution is to neutralize any data the generative AI can use to assist in designing the optimal attack.
This can only be done by assessing human risk and correlating it with threat intelligence and the exposed infrastructure to prioritize preventive measures. It means shifting some focus from detection and response to prediction and prevention. By identifying and measuring the risk of employees with privileged financial and technical access, organizations can extend and expand their defense in depth, reaching beyond their endpoint security, server security, cloud security, and cyber awareness programs, and getting closer to the source of the attacks with the objective of disrupting attackers while in the reconnaissance and resource development stages.
How Picnic Can Help
Picnic offers a frictionless cybersecurity solution that protects against social engineering and credential compromise attacks. Picnic proactively and continuously disrupts attacker reconnaissance and resource development, effectively reducing organizational risk by 65% in the first year. It does it by offering a program that uses its own proprietary technology to deliver security outcomes via managed services. With Picnic, you can disrupt generative-AI social engineering and credential compromise attacks by:
- Continuously reducing and neutralizing exposed PII and other sensitive personal information likely to be fed into AI for use in an attack.
- Continuously monitoring for compromised credentials across work, personal, and service accounts and automatically blocking their reuse within your organization.
- Continuously monitoring for, identifying, and blocking suspicious domains and accounts before they can be used for social engineering or credential harvesting.
- Personalizing security education to combat real-world threats, including AI generated campaigns, with data-driven risk-based social engineering training and advanced spear-phishing simulations.
- Protecting your high-value targets, employees, contractors, and infrastructure from being successfully targeted or exploited by threat actors using generative AI.
If you would like to learn more about Picnic and how we can help address the threat of generative AI social engineering and credential compromise attacks, schedule a demo of our services today. If you are still skeptical about the power of generative AI to influence human behavior, then be advised that this blog was co-authored by one.
Become a Subscriber to receive timely articles on human-centric security issues: