Shown an AI visibility number? Ask how many times, and with which questions
AI answers change every time you ask, so a number like "you appear in X% of AI answers" depends heavily on how it was measured. Three questions to ask whoever hands you that number, and the measurements behind them.
"Your company is named in X% of AI answers." If you are looking at AI visibility services, you will see numbers like this more and more. We produce numbers like it too.
But the number depends heavily on how it was measured, because asking an AI the same question twice can get you two different answers. Your name appears in one and is gone from the next. A figure built from asking each question once and a figure built from asking many times mean very different things.
What to check comes down to three questions.
1. How many times was each question asked?
Several studies outside Japan have already shown how much AI answers vary.
- In SparkToro's study, 600 volunteers ran 12 prompts through ChatGPT, Claude and Google's AI a combined 2,961 times. The odds of the same brand list coming up twice were under 1 in 100, and the authors suggest asking 60–100 times and averaging to get the real picture1.
- Conductor ran 50 repeats per prompt across 10 industries and 4 AI engines, about 14,000 calls in all, and found Gemini the least consistent of the four2.
We checked whether the same holds for Japanese B2B buying questions3. For one Japanese business-equipment category, we wrote 50 questions a buyer might ask and put each one to ChatGPT and Gemini six times. This is the share of questions where one company's name came and went across the six answers:
| Arm | ChatGPT | Gemini |
|---|---|---|
| 6× in one day | 50% | 46% |
| Once a day, 6 days | 46% | 30% |
On close to half the questions, the answer changed from one ask to the next. A number from asking once is one draw, whatever happened to come up that day. Ask how many times each question was asked, and how many of those times you were named.
2. When, and how spread out?
So should repeats be spread across days? The two rows above are the same 50 questions asked six times in one day and once a day for six days.
The difference in variance was −0.004 for ChatGPT (95% CI −0.032 to +0.025) and −0.023 for Gemini (95% CI −0.055 to +0.010). Both intervals include zero: we found no difference between the two ways of asking3. Which company was named first, and which rivals appeared alongside, were about the same too.
That fits Schulte et al., who found that variation in the sources an AI cites is in the same range within a day as from one day to the next4. Over a span of a few days at least, bunching the repeats together doesn't miss anything.
So "we measure once a day" is not reassuring on its own. What matters is how many times each question was asked within a measurement. Change over weeks or months, such as model updates, is a different matter and needs measurements spaced out over time.
3. Which questions, and how many?
The other finding: whether answers waver depends largely on the question. The 25 questions that wavered in an earlier round wavered again (variance 0.10–0.15); the 25 stable ones stayed stable (0.03–0.07)3.
So the number moves a lot with the choice of questions. A set of steady questions makes the figure look stable; a set of wavering ones makes it swing. Ask to see the question list, and why it has the size it has.
How we measure
Our answers to those three questions:
- Spend repeats where they matter. Ask every question a few times, find the ones that waver, and ask those more; ask the stable ones less. Same cost, better precision where it counts.
- Repeat within one measurement. Long-term change is measured separately.
- Grow the question set until it saturates. Keep adding questions until new ones stop turning up new findings; that sets the size of a company's question set.
- Every judgement carries its evidence. For each answer we count as "named", we keep the AI's own sentence and the sources it cited. Every number traces back to a specific sentence in a specific answer.
The definitions of our scores are public in the ZAG Standard.
Scope of this measurement
Our comparison is a pilot: 50 questions, two AIs, six days. We also looked at whether question length or vagueness drives variation and saw no consistent pattern at this size. Change over weeks or longer is outside its scope.
Sources
-
SparkToro, "NEW Research: AIs are highly inconsistent when recommending brands or products" (Rand Fishkin, Patrick O'Donnell). https://sparktoro.com/blog/new-research-ais-are-highly-inconsistent-when-recommending-brands-or-products-marketers-should-take-care-when-tracking-ai-visibility/ ↩
-
Conductor, "AI recommendation consistency analysis". 10 industries, 7 intents, 4 engines, 50 runs each, about 14,000 calls. https://www.conductor.com/academy/ai-recommendation-consistency-analysis/ ↩
-
ZAG research. 2026-10-29 to 2026-11-03. 50 Japanese B2B buying questions × ChatGPT and Gemini; six answers on one day vs one a day for six days, matched by question (n=50 questions per AI). Variance is the mean, across questions, of the variance of whether the company was named (0/1) over six answers. Confidence intervals from per-question differences. ↩ ↩2 ↩3
-
J. Schulte, M. Bleeker, P. Kaufmann, "Don't Measure Once: Measuring Visibility in AI Search (GEO)", arXiv:2604.07585 (April 2026). https://arxiv.org/abs/2604.07585 ↩