New Research Proves Measuring ChatGPT Responses at API-Level Only Produces Unreliable AI Visibility Data
The paper shows that API-only AI visibility tools can be unreliable data sources because citations vary across models and thinking.
Matt Southern's Search Engine Journal report covered the results of an arXiv preprint by the National University of Singapore and Xi’an Jiaotong University. The paper is titled, “Understanding ChatGPT Citations: Evidence from Production Web Search Across Model Generations”.
The paper’s main findings were:
-
Sources retrieved more frequently across repeated runs are more likely to be cited.
-
Generative engine optimization (GEO) should distinguish retrieval visibility from citation visibility
If you didn’t know, an API is technology that lets any two pieces of software talk to one another. It stands for Application Programming Interface. One of the most widely recognized ones in marketing is called a REST API, which you may have come across if you work in SEO or web development.

An image from the study (Figure 1).
Methodology
This study's methodology is what makes it so illuminative.
The researchers examined how the process of deciding which sources to cite varies by model and thinking level, measuring variance across four ChatGPT Web settings, including Instant and Thinking modes. The paper measured OpenAI Responses API and SERP API citations compared to web UI citations. The data was collected between March and July 2026, so SERP scraping poisoning doesn’t skew the results.
Jiang et al. didn’t only examine models’ final citations. Rather, they analyzed server-sent event traces (SSEs). SSEs help us observe how ChatGPT searches for information, retrieves candidate webpages, and selects references.
The researchers captured execution logs across several ChatGPT web settings, from GPT 5.2 to GPT 5.5, to see how it looks for information before it starts answering. The paper breaks down how ChatGPT’s observable search and citation-selection process determines which sources appear in final citations.
Multi-URL Support
The researchers ran the same prompts 20 times and found that cited URL sets vary substantially, while hostnames and entities are more consistent. It retrieves different pages from the same hostnames (websites/brands) or about the same entities.
However, domain footprints are stable. Having multi-URL support (multiple URLs from your website show up during retrieval) is an enormous predictor for being featured in a final citation over multiple queries. This is because LLMs operate on mathematical probabilities within their neural network.
More Articles About Facets of a Query = Higher Citation Likelihood
When ChatGPT processes a query, it uses probability to generate its next token. Multiple articles confirming certain facts create statistical consensus that something is true. So, one 5,000-word article that covers all facets of a query won’t increase your likelihood of being cited, as it’s just one data point.
Multiple articles, covering multiple facets of a problem or query, are more likely to get you cited, since they’re multiple data points all confirming the same facts. When an LLM retrieves multiple articles from your domain, all covering the same facts, the statistical weight of your entity (brand/thing/person) being cited increases in its temporary memory context window.
As a brand, when you create content that covers multiple angles of problems you solve, you mathematically reinforce the consensus that you’re an answer and build up your street cred.
More Entity/Brand Mentions Across the Web Increases the Likelihood of Generation
When an LLM sees your brand mentioned across multiple URLs, it becomes more likely to produce a citation token for your brand than another source.
APIs Don’t Allow Us to Examine What End-Users See
You can’t see how visible you are on ChatGPT with just the OpenAI Response API or SERPAPI. APIs don’t always reflect what users see when they’re using the web UI (the user interface on the ChatGPT website).
When researchers combined Google Search results with all the source candidates retrieved through the OpenAI Responses API, they recovered only 28.3% of URLs cited by ChatGPT 5.3 Instant and 26.2% of those cited by ChatGPT 5.5 Instant.
In other words, even combining Google Search with API-level retrieval data missed over 70% of the exact URLs selected for citation by ChatGPT Web.
Retrieval Visibility vs. Citation Visibility
The responses API is very effective at showing us which sources are likely to be included in an answer’s citations. However, the API’s pool of source URL candidates is very different from the final collection of citations a user sees because it’s measuring the wrong part of the process.
The researchers found that examining what the OpenAI Responses API cites internally doesn’t let us accurately predict what ChatGPT shows end users.
External proxies recover cited hostnames considerably more reliably than exact URLs, suggesting that website-level visibility is more consistent across measurement methods.
Prompt Execution and Multi-Round Retrieval
This brings us to how AIs execute prompts. When you prompt ChatGPT, it doesn’t just Google it for you.
It performs something called query fanout. This is when it rephrases your query in enough different ways that it’s able to get sources for and against the answer it would give you. Your prompt is broken into various distinct search paths, which vary a lot between model settings.
For instance, the ChatGPT Instant settings on models like 5.3 and 5.5 produce mostly single-round runs. ChatGPT thinks once, then searches. In over 31% of runs, ChatGPT 5.5 issues multiple queries.
The study also found that ChatGPT 5.4 thinking setting performs around 11.82 executed queries across 3.2 distinct retrieval rounds per one user prompt. Most of the runs on this setting use the web search tool more than once.
In later retrieval rounds, ChatGPT uses the “site:” search operator over half the time, narrowing its sources and using exact phrases much more often.
The Candidate-to-Citation Funnel
In the paper, the researchers formalized what they call the “candidate-to-citation funnel”. This is the narrowing net cast for web searchers to meet queries.
To properly examine this, they broke down ChatGPT’s retrieval windows into sets:
-
WRet: Retrieved main page (the primary URLs ChatGPT fetched).
-
WSup: The supporting pages fetched, often subpages and contextual links, attached to those main pages.
-
WCite: This represents the citations used in the final answer.
To figure out which predictive signals move a page from the WRet bucket to the WCite bucket, the researchers used a statistical method called a leave-one-run-out design. For every 20 parallel runs per query, they isolated one run and tried to predict whether a source would be cited based on performance in the other 19 parallel runs.
They found that the clearest predictor of a source moving to the bottom of the funnel was cross-run recurrence. Sources that appear frequently across runs are more likely to be cited.
The Takeaway for Brands
This research illustrates that if you want to be cited:
-
You must have brand/entity mentions of other relevant sources across the web.
-
You should produce multiple pages that answer different aspects of queries you’re relevant to and have consistent information.
Overall, if your sources appear across multiple retrieval runs, it shows your brand is reliable and should be cited. It’s no longer enough to be the right answer, you have to be the most consistently found answer.
Where Possible, OpenLens Uses WebUI Infrastructure Instead of APIs to Accurately Predict Responses
At OpenLens, we use Web UI infrastructure to collect data from the web UIs of ChatGPT, Gemini, and Google AI Overviews. If you would like access to SSEs for a deeper look at the actual sources used in OpenLens’ more advanced technologies, get in touch.
OpenLens enables you to track visibility across seven AI platforms: ChatGPT, Claude, Gemini, AI Overviews, Perplexity, Grok, and DeepSeek. For ChatGPT, Google AI, and Gemini, we show you responses as they appear in the consumer UI, ensuring information is as accurate as possible.
For Grok, Claude, and DeepSeek, we use the public API. This means we see the same responses a developer would see, plus search context where the API exposes it. When we can use the chat surface, we do.
Track Visibility Over Time
OpenLens lets you track visibility over time so that you can demonstrate incremental lift through your efforts. See how your visibility rises month by month, or even day by day. With OpenLens, you can see visibility over the full year, not just one month.
View Specific Gaps by Platform
OpenLens lets you see visibility by platform and suggests how you might close the gap. For instance, you might be more visible in Claude because of PR coverage, but less visible in ChatGPT because your vendor docs need updating.
Sign up for OpenLens to get accurate, reliable visibility data about responses shown to consumers. Track your first three brands for free and enjoy content and site audits.