Ask twice, get two answers
Open ChatGPT, ask “which term plan is best for a 30-year-old in India?”, and note the brands it names. Open a new chat and ask exactly the same thing. There’s a good chance the list is different: a brand that was first is now third, one has dropped out, a new one has appeared.
This surprises a lot of marketing teams, because search engines trained us to expect one fixed list of results for one query. AI answers don’t work that way. That isn’t a bug, and it isn’t the tool being unreliable. It’s how these systems are built. It does mean that a single screenshot of a single answer tells you very little about where your brand stands.
There are three reasons answers move.
Reason 1: the model picks its words with some chance in it
A language model writes an answer one piece at a time. At each step it has a set of likely next words and picks one, with an element of chance. Small differences early in an answer, such as which brand it happens to mention first, change everything that follows.
OpenAI says this plainly in its own developer documentation: its models are non-deterministic by default, meaning outputs may differ from request to request. Even its “seed” setting, which developers use to make answers more repeatable, is described as best effort: OpenAI says identical outputs can’t be counted on even then.
So even if nothing else changed, the same question would still produce different answers.
Reason 2: the search step changes too
When ChatGPT searches the web, it doesn’t only rely on what it learned in training. OpenAI says ChatGPT search uses third-party search providers, as well as content provided directly by partners, and shows links to the sources it used.
Every part of that chain can move between two asks:
- The searches it runs. The model decides what to search for, and it can phrase those searches differently each time.
- The results it gets back. Search results change as pages are published, updated or re-ranked.
- The pages it actually reads. It may open different pages from those results.
- Whether a page can be read at all. A site that blocks OpenAI’s search crawler, OAI-SearchBot, is not shown in ChatGPT search answers, according to OpenAI’s crawler documentation. The same page notes that robots.txt changes can take about 24 hours to take effect.
Different pages in means different brands out. This is why the websites an answer cites matter as much as your own site.
Reason 3: you and your buyer aren’t asking the same ChatGPT
The answer you see on your own phone isn’t necessarily the answer your buyer sees. People ask with different wording, at different times, in different chats.
Measurement tools add one more layer. Citationly, for example, records answers from OpenAI’s model through its API with web search on, not from the ChatGPT app. The app is a different product, so your own ChatGPT may answer differently. Any honest measurement says which model answered, on which date, and whether web search was on. If a tool doesn’t tell you, ask.
What this means for measuring
If answers move, a single ask is a sample of one. Here’s how much each approach tells you:
| What you do | What it tells you |
|---|---|
| Ask one question once | What one answer said, once |
| Ask one question three times | Whether you’re named every time, sometimes, or not at all, for that question |
| Ask a fixed set of questions three times each | How many of all those answers name you: a count with a total |
| Repeat the same set weeks later | What changed between two dates, on the same questions |
The jump that matters is from “an answer” to “a count of answers”. “We were named” is an anecdote. “Named in [n] of [N] answers we recorded” is a measurement, because anyone can see what was counted.
How to measure anyway
You can do a basic version of this by hand. The method is the same one Citationly uses.
- Fix your questions. Write the questions a buyer in your category might ask, from first looking to comparing to choosing. Keep the exact wording, and don’t change it between runs.
- Ask each question more than once. Three times is a practical minimum. Use a fresh chat each time so earlier answers don’t carry over.
- Record everything. Save the full text of each answer and the sources it cited, with the date. A count you can’t trace back to the answers is a count nobody can check.
- Count, don’t score. For your brand, count the answers that name you, and write it with its total: named in [n] of [N].
- Read each question’s three asks together:
- Named in all three: a settled position for that question.
- Named in none of the three: a settled gap. Say “not named in any of these three answers”, which is what you know, rather than “ChatGPT doesn’t know us”.
- Named in one or two: unsettled. You’re on the edge of the list, and this is often where effort pays off first.
- Repeat on the same questions. Comparing two runs that asked different questions compares nothing. And two runs show a difference, not a direction or a cause.
Why Citationly asks three times
Three asks per question is a deliberate trade-off. One ask can’t tell a settled answer from a lucky one. Many more asks per question cost more and add little for a buyer deciding where to act. Three asks show, question by question, whether your place in the answer is settled or still moving, and every count in the report keeps its total beside it. The method page of every report says how often the three asks agreed, so you can see how much the answers moved in your category.
How Citationly helps
A Citationly report asks the questions a buyer in your category might ask, three times each with web search on, and records every answer word for word, so each count opens to the answers behind it. See how it works, or book a demo to see a real report.