Executive Overview: The Transparency Gap
For SEO professionals, content creators, and researchers, the "black box" of AI citations has long been a source of both frustration and curiosity. How does ChatGPT decide which websites are authoritative enough to merit a citation, and why do these citations fluctuate so wildly? A groundbreaking preprint paper released on October 11, 2026, by researchers from the National University of Singapore and Xi’an Jiaotong University, provides the most comprehensive empirical look yet at the underlying mechanics of ChatGPT’s source selection process.
The study, which tracked data flow between March and July 2026, analyzed four iterations of ChatGPT—from GPT-5.2 Instant through GPT-5.5 Instant. By monitoring live, signed-in user sessions, researchers uncovered a significant discrepancy: the web application’s citation behavior is not easily replicable via standard search APIs or Google’s organic results. In fact, when researchers attempted to mirror ChatGPT’s source-gathering process using the OpenAI Responses API and Google Search, they could only account for roughly 26% to 28% of the specific pages cited by the chatbot.
This study confirms what many have long suspected: ChatGPT’s "Search" functionality is not a simple wrapper for a search engine. It is a highly opinionated, constantly evolving retrieval-augmented generation (RAG) system that frequently rewrites user prompts, defaults to English even for non-English queries, and exhibits high levels of stochastic variance—meaning the same question asked twice can yield entirely different sources.
Detailed Chronology: Tracking the AI’s Writing Process
To understand how ChatGPT arrives at its citations, the researchers took an "under-the-hood" approach. Rather than relying on the output alone, they recorded the real-time data flow of the ChatGPT web application during the model’s writing phase. This allowed the team to map the entire lifecycle of a search: the original user prompt, the internal search queries generated by the AI, the resulting search engine results pages (SERPs), and finally, the selection of sources.
The study utilized 80 distinct prompts sourced from Search Arena, a public repository of real-world interactions with AI models. These prompts were balanced equally between English and Chinese and spanned four critical intent categories: recommendations, fact lookups, information synthesis, and complex analysis. Each prompt was executed 20 times across GPT-5.3 and GPT-5.5 models to measure consistency.
The "Rewrite" Phenomenon
One of the most significant findings involves how ChatGPT handles user input. OpenAI’s documentation admits that the model transforms user queries into search terms, but the researchers found this happens with extreme frequency. In GPT-5.3 and GPT-5.5, the model never utilized the exact wording of the original user question. Instead, it interpreted the intent and reformulated the search to suit its internal logic.
Even more striking was the "Language Bias" observed in the data. When researchers provided Chinese-language prompts, the models frequently bypassed Chinese-language search indexes entirely, opting instead to perform searches in English (67% for GPT-5.3 and 69.6% for GPT-5.5). This suggests that even as models become more multilingual, the underlying retrieval mechanism remains heavily anchored in English-language corpora, potentially limiting the breadth of sources for non-English users.
Supporting Context & Metrics: The Replication Challenge
The researchers sought to determine if they could replicate ChatGPT’s cited sources using external tools. They used two primary methods:
- Google Organic Search: Fetching up to five top-tier results per query via SerpAPI.
- OpenAI Responses API: Executing the same queries through the official API with web-search capabilities enabled.
The "Page vs. Site" Discrepancy
The data revealed a stark contrast between the predictability of sites versus pages. When comparing the citations provided by the ChatGPT web app against the results gathered by the API and Google, the overlap was minimal at the individual page level but significantly higher at the domain level.
- Page-Level Accuracy: Only 28.3% of pages cited by GPT-5.3 and 26.2% of pages cited by GPT-5.5 could be accounted for by the combined efforts of the Google API and OpenAI’s own Responses API.
- Site-Level Accuracy: When looking at the root domains, the accuracy jumped to 50.3% and 48.7%, respectively.
This suggests that while ChatGPT may have a "preference" for certain domains or news aggregators, the specific article or page it selects is often a result of internal post-processing—a layer of the AI’s "Thinking" or "Instant" logic that standard search retrieval does not capture.
The High Cost of Variance
Perhaps the most concerning metric for SEOs is the lack of repeatability. For a single question, 44.5% of the distinct pages cited by GPT-5.5 appeared in only one of the 20 test runs. This confirms that the model’s citation strategy is highly fluid. The authors noted that while pages appearing in more runs had a higher likelihood of being cited, this was not a guarantee. Citation behavior appears to be influenced by a complex interplay of the model’s current internal state, the specific "search query" it generated, and the real-time availability of content at the moment of the request.
The Evolution of the "Black Box"
The research highlights a critical instability in AI-driven search. During the data collection period, the transition from older models (GPT-5.2) to newer ones (GPT-5.5) resulted in a massive shift in how information was retrieved and prioritized.
For example, when OpenAI updated the underlying model to GPT-5.5 in May 2026, third-party data from sources like Sistrix showed that citation variance—the day-to-day fluctuation in sources provided—spiked from a nominal 1–2% to nearly 47% in some markets. This implies that any SEO strategy built on "ranking for AI" is playing a game with moving goalposts. If the model changes, the citation logic changes with it.
The researchers explicitly warn that standard peer review—which takes months—is insufficient for evaluating these systems. Because the "AI landscape" shifts every few weeks, by the time a study is published, the models being tested may already be deprecated.
Official Implications & Future Outlook
The study concludes with a series of recommendations for those looking to monitor their brand’s presence in AI citations.
Recommendations for Monitoring
- Monitor at the Domain Level: Because page-level citations are erratic, organizations should focus on their domain’s overall inclusion in the AI’s "consideration set" rather than obsessing over individual article rankings.
- Account for "Query Rewriting": SEOs must recognize that their target keywords may be completely replaced by the AI during the search phase. Tracking what the AI actually searches for (by monitoring search logs or simulated API interactions) is more important than tracking the user’s original prompt.
- Perform Multi-Run Testing: A single query is insufficient to determine visibility. To understand how often a brand is cited, researchers suggest at least 20–50 runs per query to establish a baseline probability.
The Road Ahead: GPT-6 and Beyond
As of late 2026, the industry is transitioning to GPT-6, which is currently rolling out to Enterprise and Plus users, and GPT-5.6 "Luna," the new standard for the free tier. The researchers note that their study provides a "snapshot in time" of the GPT-5.x era, but the methodology remains the blueprint for the future.
The fundamental issue remains: as long as OpenAI keeps its retrieval and ranking algorithms proprietary, the "why" behind these citations will remain speculative. However, the data confirms that ChatGPT is not merely summarizing the top results of a Google search. It is an autonomous agent that decides, based on opaque internal heuristics, which parts of the internet are "worth" referencing.
For the digital marketing industry, the lesson is clear: the era of static search rankings is dead. In the age of AI, visibility is a transient, probabilistic, and highly dynamic phenomenon that demands new tools, new metrics, and a fundamental shift in how we perceive the relationship between a website and a machine-generated answer.
Note: This article is based on the findings from the arXiv preprint "Automated Evaluation of LLM Citation Patterns," authored by researchers at the National University of Singapore and Xi’an Jiaotong University. The study has not undergone formal peer review.
