How AI Phishing Is Going Right Past Your Software And Training

It seems like every day now there is a new cybersecurity breach, hack, or large scale attack that steals credentials and millions of dollars. Based on data from Palo Alto's Unit 42, their #1 most analyzed threat from 2024 was phishing.

With the introduction of AI, hackers now are equipped with a capability that can make malicious emails, at scale, that can seem like normal emails you click on every, single day.

So what can we do? If you look to CISA or other authoritative industry organzations they'll tell you that training is the best way to keep users and companies safe. Well I hate to break it to CISA, but there has been training for years and people still take risky chances with emails and links at an astonishing rate of 94%.

Let's take a look at the current state and some of the more recent AI phishing attacks and see if you would have been capable of noticing them...

The Current State Of Phishing

Back in the day, phishing was the stuff of comedy and amateur hour. There were memes about a "Nigerian prince" asking for your PayPal, or a poorly worded email that seemingly came from a family member. Most people ignored these very easily and very few were successful or beneficial to the hacker.

Then you had APTs come along, looking to up the game and target very powerful people with more sophisticated messaging. These were not common, and the capability was not widely spread or used against a large group of people so the threat was very focused. As long as these people had things in place to help mitigate the attack after a user clicked on a link or an attachment, they were safe.

Then we fast forward to today, where anyone can go to any number of online AI chats and request emails in business professional language for any number of topics. These include business proposals, IT Security check ups, password resets, and many more. All hackers have to do is swap the link for their malicious one and now they have a powerful tool. Things get even scarier when we talk about creating thousands of malicious emails in a few minutes using these AI tools.

The days are over of phishing emails having dodgy sender info too, which was a dead give away in older campaigns. In some cases, phishing emails are even impersonating enterprise companies like Microsoft or Google to gain your trust. People look at the sender info and the language and think "okay, this must be from the legit company" when in reality it was generated using AI.

Let's take a look at some specific examples where AI has been used to create some extremely compelling phishing campaigns and you be the judge: would you have been able to spot it before it was too late?

Case #1: Bill Sempf- POINT BEC Attempt

On March 12th 2025, Bill Sempf posted a blog post titled "Weird unsolicited email of the week."

This case is really interesting because the email itself is extremely well written and seems like a genuine proposal to the intended recipient.

Here's the email that was sent:

The above email appears completely legitimate, but was actually AI-generated and part of a Business Email Compromise (BEC) attempt. If you happened to mouse over the link, or just copy it and paste it into a notepad or other editor, you would have seen this:

Notice the redirectTo parameter, which takes you to a weird calendly link. Now, there isn't much in terms of magical malware being downloaded here, but a much more sophisticated attack chain.

We've seen in recent months a novel attack method using zoom calls to gain access to secure accounts, credentials, and other sensitive platforms. What the attacker does is when the victim joins the call, their video and audio is turned off. The attacker then types in the chat with a link to their own video conferencing platform that they are also logged into and their video/voice works just fine. Then, without even second guessing it the victim clicks the link and any number of compromises can occur.

Now, if you're sitting there thinking that you would never be targeted by an email like this I want you to think again...

With AI, anyone has the capability of doing some quick recon on your social media and figure out how to craft thousands of email or DMs that will get you to click a link. The question is, will you be able to spot it?

Case #2 Zach Latta - Google AI Phone Call

We've already written on this case before, but we'll give a summarized version here as it applies to the state of AI and phishing.

Zach wrote a blog on his github that outlined a very sophisticated attack using AI. The attacker called him from a Google number (more on this later) and claimed that his account was likely compromised and they needed to verify his identity in order to unlock his account.

He agreed and the attacker sent an automated email with the details on how to get his account back.

The voice was "rather human" and seemed to be legit, a female security specialist from Google. When Zach became skeptical, the AI was able to calm him down and assure him that everything was going exactly as planned.

It wasn't until the voice on the call sent him a text for verification that he realized something was off- the link they sent looked strange.

Now, the email by itself is very impressive and most people would have clicked a link from that email right off the rip. But the AI phone call with a full on conversation? The automated text with the ability to take objections and handle it perfectly fine?

This is where we are at with AI and phishing and it's almost impossible to believe the average person, or the successful business owner, will be able to stop it.

Case #3 IBM Generated AI Emails

This case follows a test that IBM ran against their employees in order to see how effective AI generated emails were at compromising employees (spoiler: they're really effective).

With roughly 5 prompts, the IBM X-Force analysts were able to generate "phishing emails tailored to specific industry sectors".

Here is an example of one of the emails below:

It was sent to 800 employees at the company with a click through rate of 52%. That's over 400 employees compromised from one phishing email...

It is very plausible that any attacker could follow the same method to create thousands of phishing email templates and send them out to large email lists to try and get anyone to click. Think free coupons to your favorite businesses or fast food restaurants just before dinner time, or a huge sale on the items you like to get and post on your social media.

There are many ways to figure out what makes any person want to spend money or attention and have AI generate emails just for that.

What You Can Do

From these examples, it can feel a bit overwhelming if you don't know how to verify links like a cybersecurity pro. There are plenty of free resources and techniques that I'll outline below that can help you if you don't know how to identify phishing, even the sophisticated ones above.

• Keep an eye out for easy signs like domains that don't match the email (A security email from Facebook but the link is for a random .xyz site)
• Verify that you even have an account with the site that sent you the email, sometimes users will click links that they don't have have an account with
• Mouse over a link (on your laptop or desktop) and it should show the link in a preview in the bottom left side of the browser
• Copy and paste the link into Virus Total or URLscan.io - both of these tools are used by cybersecurity professionals when investigating

All of these techniques can help you stay safe even when emails look very safe. When in doubt, go to the website itself and reach out to support to verify that the email is legitimate before clicking anything or responding. It's best practice to go to the known good source for more information then to click any links in the email.

If you need something that can detect and remove phishing links automatically for you, we developed a tool for that exact purpose: RedPhish.

You can check it out on our homepage and you can download it from the Chrome Web Store on Chrome, Opera, and Edge. We are working on getting it onto iPhone with Safari App Extensions so stay tuned for that.

Feel free to connect with us on social as well, we are on Facebook, Instagram, Linkedin, and X.