Google Is Feeding SEO Tools Fake Results. Here’s How to Tell If Your Rankings and AI Visibility Reports Are Wrong

Your report says you fell off page one. Your phone still rings. Both can be true right now. Here is what is going on, how to tell the difference, and what to ask before anyone touches your website.

What happened, and when

A rank tracker asks Google for results pages, over and over, and reads where your site sits. Since mid-September, some of those pages have come back altered.

Nozzle, a rank-tracking company, calls them “poisoned”. A poisoned page has swapped listings. It does not match what a person would see.

Here is the timeline, from the vendors’ own reports:

• From September 13. Nozzle counts 8.0% of the pages it collected as poisoned, against 0.17% in the weeks before.
• September 26. SEOmonitor sees ranking changes that people searching Google did not see, and blames incomplete data sent to automated tools.
• September 27 to 29. The worst days. Nozzle says poisoned pages peaked at 27.0% of everything it collected.
• From October 1. Roughly 5% to 11% of pages a day.
• October 7. DemandSphere reports that Google returns spurious results when it suspects bot traffic, about 11% of its daily results pages, as PPC Land covered.
• October 10. Nozzle publishes its study of 27.5 million results pages in 49 countries, summarised here.

Search Engine Roundtable reports Nozzle’s explanation of what Google is doing: blocking very wide ranges of IP addresses, serving far more captchas, and injecting bad data into pages it suspects are bots.

Google has not commented. Treat every figure as provisional. They come from vendors measuring their own data, and the vendors say the numbers are still moving.

What a false drop looks like in your report

On a poisoned page, Nozzle found a tracked site was missing from the top 10 in 58.6% of checks. On a normal page, that happens 2.7% of the time.

The same study found 77.9% of those drops recovered within three days. That is the pattern to look for: a sharp fall that fixes itself.

Why Canadian businesses should care

Nozzle’s country numbers put Canada at 10.5%. That is third among the larger markets, behind Australia and India. It sits above the United States at 8.5% and the 8.0% average across all 49 countries.

Nobody has said why. Nozzle also found that local targeting on its own made no difference. So the data does not show local service businesses being hit harder than anyone else.

It still matters in Toronto. Local businesses lean hard on rank reports. A plumber, a dentist or a restaurant often tracks 10 to 30 phrases for the city.

On a list of 20 phrases, 10.5% works out to about two. That is simple arithmetic on Nozzle’s average, not a measured result for your keywords. It shows how a few false moves can land in a single report.

What it may do to AI Overview and AI visibility reports

None of the reports above tested AI visibility tools. So this section is reasoning, not data.

Many tools that record Google’s AI Overviews and AI Mode answers collect them the same way trackers collect rankings: with automated requests. If Google is blocking, challenging or feeding bad data to automated traffic, those tools could be caught too. Nobody has confirmed that.

AI answers also change from run to run when nothing is wrong. We covered that in how to measure AI visibility when ChatGPT keeps changing. So a swing in your AI visibility score now has two possible causes: normal variation and bad collection. Neither one means customers stopped finding you.

Treat a one-week drop in an AI visibility score as a question, not a verdict. Look for the same drop in visits from AI tools in your analytics. If the score fell and the visits did not, wait.

The questions further down apply to AI reports too. Our AI search optimization service page shows how we scope this kind of work.

Five signs a ranking drop is false

One sign alone is not proof. Look for two or three together.

1. Many pages drop at once. A real penalty or a competitor’s move rarely hits unrelated pages and phrases on the same day. A data problem can.
2. Search Console is flat. Nozzle’s own advice: if Search Console still shows impressions and clicks, nothing happened to your site.
3. The phone and inbox look normal. Calls, directions and enquiries are holding steady.
4. You can’t find the drop yourself. Search the phrase in a private window. Barry Schwartz of Search Engine Roundtable says spot-checking a few queries in an incognito window is the only way he trusts a movement right now.
5. It bounces back. On poisoned pages, 77.9% of drops recovered within three days. A real loss usually stays.

The numbers to trust instead

Rank is not revenue. Start from the money and work backward. We wrote about this in the marketing metrics that actually matter.

• Bookings and quote requests. Real people asking for work. Nothing a tool collects beats this.
• Business Profile calls, directions and website clicks. Counted by Google on your own listing, so no rank tracker sits in the middle.
• Search Console clicks and impressions. Real searches by real people, not a tool’s test requests.
• AI referral traffic. Visits from ChatGPT, Perplexity, Gemini and similar tools, in your analytics. Here is how to see how many customers AI actually sends you.

Keep your tracker. Use it for direction and for spotting something worth a look. Just don’t let it make decisions alone.

What to ask your agency or tool vendor before changing anything

If a report says rankings fell, ask these before anyone rewrites pages, builds links or moves ad budget. Ask for the answers in writing, with dates.

• Did you compare this drop with Search Console and Business Profile data? A rank drop with flat clicks needs an explanation.
• How does the tool collect its results? Ask whether it flags incomplete or suspicious pages.
• Has the vendor said anything about the problem since mid-September? Ask if data from September 13 onward was checked or re-collected.
• How many phrases moved at the same time? A broad, sudden move points to the data, not your site.
• Will you recheck in three days before we change anything? Most altered-page drops in Nozzle’s data came back by then.
• Does this report mix tracker positions with real traffic? You want both, labelled clearly.

A good partner will welcome these. If the answer to the first one is “we didn’t check”, that tells you something too.

A 10-minute Toronto check you can run yourself

You do not need a tool for this. You need a private browser window, Search Console and your Business Profile.

1. Pick three phrases that bring you customers. Choose the ones you would hate to lose, like “[your service] Toronto” or “[your service] near [your neighbourhood]”.
2. Search each one in a private window. Note where your listing sits in the map results and in the regular results. Your view can differ a little from a customer’s, so look for big gaps, not one-spot moves.
3. Match the date. Compare what you saw with what your tracker says for the same day.
4. Check the real numbers. In Search Console, compare the last 7 days with the 7 before for the same pages. In your Business Profile, compare calls and directions.
5. Repeat in three days. If the report recovers and your numbers stayed flat, log it as a false alarm. If your numbers fell too, start an audit.

Write down each check with the date. A free GEO tracker template works for this, and it also covers AI answers.

Frequently asked questions

The bottom line

Rank trackers are still useful. They are just less reliable right now than the charts make them look. Until the vendors say the problem is fixed, treat a sudden drop as a question to check, not an emergency to fix.

A tracker tells you what it saw. Your customers tell you what happened.