Why checking ChatGPT once tells you almost nothing

You open ChatGPT, ask it to recommend a business like yours, and there you are, named in the answer. It feels like proof that AI knows you. Ask the same question again an hour later and your name can be gone, with two other businesses in its place. Nothing about your website changed. The tool simply gave a different answer, because giving a slightly different answer each time is how it works.

The check that feels like proof

Most owners meet AI search the same way. They type one question into ChatGPT, read one answer, and draw one conclusion. If they are named, they relax. If a competitor is named instead, they worry. Either way they are reading a single roll of the dice as if it were a settled score.

The problem is not that the check is hard. It takes a minute. The problem is that one answer, on its own, carries almost no information about how you usually do. A business named once out of ten tries and a business named nine times out of ten can produce the exact same screenshot on a good day.

Why the answer moves

An AI assistant does not keep a fixed list of the best businesses in your area and read it back. Each time you ask, it builds a fresh answer, choosing the next word by what is most likely to fit, with a deliberate bit of randomness mixed in so its writing does not sound wooden. That randomness is a feature. It is also why the same question, asked twice, can pull different sources, name different businesses, and put them in a different order.

On top of that, the assistant may fetch live pages while it answers, and what it fetches can vary from one run to the next. So even the raw material behind the answer shifts. The result is a tool that is confident every time and consistent almost never.

One check is a coin flip, not a verdict

Think about what a single check can and cannot tell you. If you flip a coin once and it lands heads, you have not learned whether the coin is fair. You have learned that this one flip was heads. A single AI answer is the same. It is one outcome from a process that has a range of outcomes, and the one you happened to see says little about the rest.

This cuts both ways, which is the part worth sitting with. A lucky check can make a real visibility problem look solved. An unlucky check can make a business that AI names most of the time look invisible. Either way you act on a number that was never stable enough to act on.

Measure a rate, not a moment

The fix is the same one a pollster uses. You do not ask one person and call the election. You ask many, and you report a rate. For AI visibility that means asking the same question several times and writing down how often you were named, as a fraction: named in four of ten runs, or eight of ten, or zero.

That fraction is the thing worth tracking. It is stable enough to compare against last month, and against the competitors who keep taking the spots you want. A screenshot cannot do either of those jobs. A rate can. Our guide on how to check if ChatGPT recommends your business walks through the exact questions to ask and how to score them.

How many times is enough?

Two runs is not enough. Because the answer carries built-in randomness, two runs can agree by luck and hide how much the result really moves. A handful of runs per question, per engine, gives you a fraction you can start to trust. The closer your rate sits to the middle, named about half the time, the more runs you need before the number stops jumping around, because that is exactly where the noise is loudest.

You do not need to be precise about the statistics to get the discipline right. Ask each question at least five times. Spread the runs across different days rather than firing them back to back, so a single browsing hiccup does not colour the whole result. Then report the fraction, not the best run you saw.

Do it for each engine on its own

ChatGPT, Claude, Perplexity and Google's AI answers do not think alike, and they do not agree with each other any more reliably than each one agrees with itself. A business named often in one can be absent from another. Averaging them together hides the gap that matters, so keep a separate rate for each engine your customers actually use. When you later work on the problem, you will know which engine moved and which did not.

No honest check ends in a guarantee. Nobody can promise ChatGPT will name you, because the tool does not promise it to itself. What a proper measurement gives you is the truth about where you stand today, and a number steady enough to tell whether your work is moving it.

What to do with the number

Once you have a rate per engine, the rest follows. Re-run the same questions monthly and watch the fraction, not any single answer. Score two or three competitors in the same runs, so a change in your rate is read against a moving field rather than in isolation. When a rate is low, that is the signal to look at the barriers underneath it, which is what an AI visibility audit is for.

The single check is not useless. It is a fine way to get curious. It is just a poor way to decide anything, because it was one roll all along. Measure the rate, watch it over time, and you trade a feeling for something you can actually manage.

Written byDima SmetaninCo-founder, SEENMETRIC