Prompt tracking can tell you how often a brand or a domain appears when a chosen set of prompts is run many times against an assistant. It cannot tell you where you “rank,” what any real person saw, why an appearance changed, or what to fix on a page. The first is a measurement of a distribution. The others are claims the method has no way to support, and most of the confusion around AI visibility reporting comes from reading the first as if it were the others.
This post is about the method, not about any product that implements it. It covers how the technique works, the three properties that limit it (run-to-run variance, synthetic prompts, and personalization), what published research says about each, and a short list of questions to ask of any report before it goes in front of an executive. It sits under How to audit a site for AEO and GEO, which places this kind of measurement beside the page-side work you control.
What is prompt tracking?
Prompt tracking is the practice of sending a fixed list of prompts to one or more assistants on a schedule, storing the answers, and counting which brands, products, or domains appear in them. The output is usually a share-of-voice figure, a visibility percentage, a citation count, or a position within a list.
Every implementation makes the same four choices, and each one constrains what the numbers mean:
| Choice | What it decides |
|---|---|
| The prompt list | Which questions are being asked, and in whose words |
| Runs per prompt | How many samples stand behind each number |
| The client | API or consumer interface, logged in or not, which location, which memory state |
| The counting rule | What counts as an appearance: a mention, a link, a citation, a position |
None of these is wrong to choose. They are just choices, and a report that doesn’t state them is reporting a number without its denominator.
Can one run of a prompt tell you anything?
Barely. Assistants generate a new answer each time, and published testing shows the answers to identical prompts vary more than most dashboards imply.
The largest public test of this is SparkToro’s January 2026 study. In it, 600 volunteers ran 12 different recommendation prompts through ChatGPT, Claude, and Google’s AI Overviews or AI Mode a combined 2,961 times, with each prompt run 60 to 100 times. The finding: there was less than a 1 in 100 chance that ChatGPT or Google’s AI would return the same list of brands in any two responses out of a hundred, and ordering was closer to 1 in 1,000. The number of items in each list varied too (SparkToro, 27 January 2026).
The same study is careful about what that doesn’t mean. Across many runs, some brands appeared consistently: one agency appeared in 85 of 95 Google AI responses to the same prompt, and a hospital appeared in 69 of 71 ChatGPT answers while being the first item in only 25 of them. The authors’ conclusion was that visibility percentage across many runs is “(probably) a reasonable” metric, and that a “ranking position in AI” is not.
An academic paper published in April 2026 reaches the same place from the other direction. It argues that because answers “can vary across runs, prompts, and time,” one-off observations are unreliable, and that visibility should be characterized “as a distribution rather than a single-point outcome” (Schulte, Bleeker, and Kaufmann, Don’t Measure Once, arXiv:2604.07585).
Two consequences for reading a report:
- Position is noise. If ordering repeats roughly once in a thousand runs, a “moved from #3 to #5” line is describing two samples from the same distribution.
- A week-over-week change needs a sample size behind it. A visibility figure computed from a handful of runs per prompt can move several points with nothing having changed at all. If a report doesn’t say how many runs each figure is built on, you cannot tell signal from resampling.
Do the tracked prompts represent what people ask?
Not in their wording, and the method has no direct way to know. Tracking needs a fixed list of prompts. Those prompts are written by a person or generated by a model; they are not drawn from the assistant’s own query logs, which an outside tracker does not have.
SparkToro tested how much real phrasing varies. Volunteers were asked to write their own prompt for the same underlying need; across 142 responses, the average pairwise semantic similarity was 0.081 (SparkToro, 11 February 2026). The study’s own analogy: the average pair of prompts was about as close as Kung Pao chicken and peanut butter.
Here too the research is more measured than its headline. When those 142 differently worded prompts were run, the same few brands appeared in 55 to 77 percent of the 994 responses. Assistants appear to resolve varied wording to a similar underlying intent. So the defensible reading is this: a prompt set can be a reasonable proxy for an intent if it is large and varied, and a poor one if it is a dozen “ideal” phrasings someone wrote in an afternoon.
What the method can’t recover is volume. Keyword research has impressions and search volume behind it. A tracked prompt has no equivalent number attached, so a report that weights prompts equally is implicitly claiming they matter equally, and nobody has measured that.
Does the tracker see the answer a customer sees?
Not necessarily. Assistants document personalization inputs that a tracking client either doesn’t have or holds in a fixed state.
OpenAI’s help article on ChatGPT search describes two. ChatGPT “collects general location information based on your IP address” and may rewrite a query using it, so “restaurants near me” becomes a query for a specific city. And if memory is enabled, ChatGPT “may also leverage relevant information from memories” when it rewrites a prompt into a search query; OpenAI’s example turns a request for restaurants “I’d like” into “good vegan restaurants San Francisco” for a user whose memories say they are vegan and live there.
A tracker running from a data center, logged out or with an empty memory, is sampling the answer for a user in that data center’s region with no history. That is a valid sample of something. It is not a sample of your customers, who are logged in, located somewhere specific, and carrying months of conversation. The SparkToro study lists whether API responses match what real users see in the interface as an open question rather than a settled one.
What can prompt tracking legitimately tell you?
Used carefully, three things:
- Presence within a consideration set. Across many runs of many prompts for one intent, how often does your brand appear compared with others? This is the use the published research supports best.
- Which sources an assistant cites for an intent. When answers include links, the cited domains across many runs tell you who the assistant is drawing on. That is useful competitive and content research, and it doesn’t depend on position.
- Coarse direction over long windows. A visibility percentage built on enough runs, compared month over month with the method held constant, can show a sustained shift. A single week rarely can.
What can’t it tell you?
| Question | Why the method can’t answer it |
|---|---|
| Where do we rank in ChatGPT? | Ordering is close to random between runs |
| What did our customers see? | Prompts are not theirs; location and memory differ |
| Why did visibility change? | Model updates, retrieval changes, and competitors’ pages all move the same number as your edits |
| Which page should we fix? | An appearance count has no page-level diagnosis attached |
| How much traffic did this bring? | Appearances aren’t visits; referral traffic is measured elsewhere |
The fourth row is the one that matters most for an SEO team. A visibility number going down is not a work order. The work order comes from the page side: whether crawlers can reach the page, whether the answer is extractable, whether the entity is legible. Those are deterministic properties you can check, fix, and re-check, and they are what How to audit a site for AEO and GEO is about. Whether a retrieval crawler is actually fetching your pages is a log question, covered in Reading server logs for AI bot traffic.
How should you read a prompt-tracking report?
Ask six questions before quoting any number from it. None needs a terminal; they are questions for whoever produced the report.
- How many runs is each figure built on? If the answer is one per prompt per period, treat every change as noise.
- Where did the prompts come from? Customer language from support tickets, sales calls, and Search Console queries is better evidence than prompts a model wrote about your category.
- How many prompts per intent? One phrasing per intent is a keyword-era assumption the SparkToro data contradicts.
- Which client, which location, which account state? API or interface, logged in or out, memory on or off.
- What counts as an appearance? A brand mention, a linked citation, and a first-position recommendation are different events.
- Is position reported? If so, discount it.
A report that answers all six, reports percentages with the run count beside them, and doesn’t claim to know what customers saw is a reasonable input to a business conversation. One that can’t is a set of anecdotes in a chart.
Where does page-side measurement fit?
It answers a different question. Prompt tracking samples an outcome you cannot control. A crawl measures inputs you can: access, extractability, structure, entity markup, freshness. The two belong in the same report, kept apart, with neither standing in for the other. What GEOBee measures, and what it deliberately doesn’t explains why our own tool stays on the input side and says so, and Prioritizing AEO fixes shows how to turn page-side findings into an ordered work list. For the wider question of where this kind of reporting sits in a team’s routine, see Running AEO and GEO as a practice.
If you want the input side measured per URL, with scores you can re-run after every fix, download GEOBee and crawl your site during the 7-day trial.