ChatGPT Citations Keep Shifting. Here’s How to Measure AI Visibility Anyway

If you check your brand in ChatGPT this morning and again this afternoon, you may get two different answers.

Different brands. Different order. Different sources cited underneath.

That isn’t a bug in your tracking. It’s how these systems work right now, and it changes how AI visibility should be measured and reported.

This guide covers what shifted in ChatGPT’s citations this summer, why one-off checks mislead, and a measurement routine that still holds up when the ground moves.

What changed in ChatGPT in August 2026

On August 6, 2026, OpenAI’s release notes said GPT-5.6 Luna was rolling out as the default for people on the Free and Go plans that week, while Plus and Pro subscribers got a refreshed version of GPT-5.6 Sol.

Within days, AI visibility trackers started reporting a sharp change in what ChatGPT cited.

• Promptwatch measured Reddit’s slice of all ChatGPT Search citations shrinking from 3.83% (July 18 to August 7) to 0.52% (August 14 to 17). That’s an 86.4% fall.
• OtterlyAI reviewed reports for 16 brands in unrelated industries. ChatGPT’s Reddit citations dropped 73.4%, from about 497 a day to 132. Every one of the 16 brands lost Reddit citations, and three lost all of them.

OtterlyAI also looked at what took Reddit’s place. Brands’ own official pages, and their competitors’ pages, picked up citations in 10 of the 15 reports where those pages appeared. Reference and encyclopedia sites were up in 4 of the 5 reports that included them.

For a brand that relied on Reddit threads to get mentioned, this would have looked like a visibility collapse overnight. Nothing on their own site had to change for it to happen.

Why one AI answer tells you almost nothing

Model updates aren’t the only source of movement. Answers also vary from one run to the next, even when nothing has been updated.

In January 2026, SparkToro’s Rand Fishkin and Gumshoe’s Patrick O’Donnell had 600 volunteers put the same 12 prompts into ChatGPT, Claude and Google’s AI, close to 3,000 times in total. Their findings:

• An identical list of recommendations came back twice less than 1% of the time.
• An identical list in an identical order was rarer still, roughly one time in a thousand.
• Yet some names kept coming back. For hospitals, agencies and consumer brands, certain brands showed up in 60% to 90% of the answers to the same kind of question.

That last point is the useful one.

Your “position” in an AI answer is close to random. How often you appear is not.

Semrush says the same in its own documentation. Because AI answers shift quickly and vary from person to person, it says no tool can produce exact visibility figures.

So the goal isn’t a perfect number. It’s a consistent one.

Consistent beats exact

Think of AI visibility like checking your weight. One reading after a big lunch tells you nothing. The same scale, at the same time of day, every week, shows you the real direction.

The same applies here. You can’t get a true, fixed AI visibility score, but you can measure the same way every time. When the method stays fixed, a change in the result means something changed out in the world, not in how you measured it.

That’s also what makes AI visibility reportable to a client or a leadership team. You stop defending a single screenshot and start showing a trend with context.

How to measure AI visibility when answers keep changing

This is the routine we’d recommend to any business tracking its presence in AI answers.

1. Build a fixed set of prompts

Write down the questions your buyers actually ask an AI tool before they’d contact you. Mix broad category questions with specific ones, like comparisons and “best for” questions.

Then lock the list. If you rewrite your prompts every month, you lose the ability to compare one month with the next.

2. Measure how often you appear, not where

For each prompt, record two things: whether your brand is named in the answer, and whether one of your pages is cited as a source.

Your headline number is the share of prompts where you appear. Ignore whether you were listed first or fourth. As the SparkToro research shows, order changes too often to mean much.

3. Run each prompt more than once

One run is a sample of one. Run the same prompt a few times, and in more than one tool, before you record a result.

If you show up in two out of three runs, that’s a different story from showing up once by luck.

4. Keep a log of outside changes

Next to your results, note anything that could explain a jump: model announcements in release notes, Google updates, a site redesign, new reviews or press coverage.

When your numbers move in the same week as a model update, you’ll know where to look first.

5. Compare against competitors, on the same prompts

Your visibility on its own can mislead. If everyone in your category dropped in the same week, the model changed. If only you dropped, something specific to you changed.

Track two or three direct competitors on the same prompt set so you can tell the difference.

6. Report trends over several weeks

Review results monthly and look at the direction over two or three months. Day-to-day swings are mostly noise.

When you report, explain what moved, what likely caused it and what you’re doing next. That’s a far more useful conversation than “we were mentioned 14 times.”

What to do when your AI citations drop

Before you change anything, work through these questions in order:

1. Did the model change? Check the AI platform’s release notes for the week your numbers moved.
2. Did a whole type of source lose ground? If you relied on one channel, like Reddit threads or third-party listicles, and that channel lost citations across the board, the drop isn’t about your content.
3. Did competitors drop too? A category-wide drop points to the model. A drop that only hit you points to your pages or your mentions.
4. Are your own pages clear and current? After the August shift, OtterlyAI saw official brand pages gain citations in most reports. Your service pages, pricing, About page and FAQs should state the facts plainly and match each other.

That last point connects to site structure. If your own pages are the clearest source of truth about your business, you’re less exposed when AI tools stop leaning on third-party sites. We cover how to set that up in our guide to site structure and AI visibility.

Tracking it with Semrush

You can run this routine by hand in a spreadsheet. A tool makes the daily runs, competitor comparisons and history much easier to keep up with.

Semrush One includes the AI Visibility Toolkit alongside the SEO Toolkit. According to Semrush’s documentation:

• Prompt Tracking runs your chosen prompts every day across ChatGPT Search, Gemini, and Google’s AI Overviews and AI Mode. This is where your fixed prompt set from step 1 lives.
• Visibility Overview and Competitor Research update daily on a rolling basis, so you can compare your presence with competitors over time.
• Brand Performance refreshes weekly and adds your share of voice plus the tone AI takes when it talks about you.

Daily data doesn’t mean you should react daily. Use it to build the monthly trend, and annotate it with the outside changes from step 4.

Prefer to start free? Our GEO / AI Visibility Tracker template gives you a simple way to log the same prompts each month, and our Radar scan gives you a quick first read on whether AI recommends you.

Frequently asked questions

Measure the trend, not the moment

AI answers will keep changing. Models will keep updating, and sources will rise and fall.

You can’t control that. You can control how you measure it.

Same prompts. Same method. Read the trend.

About the author

Simar Singh leads Digital 6ix, a Toronto SEO, GEO and content agency. His focus is a simple question: what do ChatGPT, Gemini, Perplexity and Google’s AI answers say about a business, and how can that be improved? Free templates from the Digital 6ix team are at digital6ix.ca/templates, and Simar shares more on LinkedIn.