Executive Overview
The mechanisms driving AI-generated citations have long remained a "black box," leaving researchers, SEO professionals, and content creators to guess how models like ChatGPT select the sources they present as authoritative. A recent preprint paper from researchers at the National University of Singapore and Xi’an Jiaotong University has finally pulled back the curtain, offering a granular look at the data flow between user prompts and the final citations provided by OpenAI’s models.
The study, which analyzed models ranging from GPT-5.2 Instant to GPT-5.5 Instant, reveals a startling disconnect: ChatGPT’s citation behavior is highly idiosyncratic and often inconsistent. When researchers attempted to replicate the model’s source selection using standard Google search results and OpenAI’s own Responses API, they found that these external tools could only account for roughly 26% to 28% of the specific pages cited by the chatbot. Even more revealing is the discovery that ChatGPT frequently rewrites user prompts into internal search queries and exhibits a significant bias toward English-language sources, even when the original input is provided in Chinese.
As AI models evolve toward the GPT-6 era, this research underscores a critical reality: standard SEO tracking tools and traditional search monitoring methods are ill-equipped to audit the unpredictable, non-linear way modern LLMs synthesize information.
A Detailed Chronology of the Research
Between March 29 and July 29, 2026, the research team conducted an exhaustive analysis of ChatGPT’s web-browsing behavior. The study utilized 80 distinct prompts sourced from "Search Arena," a public database of real-world user interactions with AI models. These 80 questions were balanced equally between English and Chinese and spanned four distinct request categories: recommendation, fact lookup, information synthesis, and complex analysis.
The Methodology of Observation
To gain an unfiltered view of the "citation process," the researchers recorded the live data flow within their own authenticated ChatGPT sessions. By capturing the underlying network traffic, they were able to witness the model’s internal decision-making process in real-time, specifically:
- Query Transformation: Observing how ChatGPT modified the user’s original prompt into a search-engine-ready query.
- Search Execution: Identifying which search engine results were retrieved by the model.
- Selection Logic: Pinpointing exactly which pages were chosen as sources, regardless of whether they were presented as individual links or bundled into group citations.
The researchers performed 20 iterations for each question on GPT-5.3 Instant and GPT-5.5 Instant, with limited runs for earlier models (GPT-5.2 and GPT-5.4 Thinking) that were decommissioned during the study period. To establish a baseline for comparison, they cross-referenced these results with up to five pages of organic Google results per query and 20 parallel queries through the OpenAI Responses API, ensuring that all external checks were performed within a 72-hour window of the ChatGPT runs.
Supporting Context & Metrics: The "Source Gap"
The core finding of the research—that standard tools capture only a fraction of ChatGPT’s sources—highlights the volatility of AI citation patterns.
Page-Level vs. Site-Level Discrepancies
The study found that while the combination of Google search results and OpenAI’s API could match roughly 26% of the specific pages cited by GPT-5.5 Instant, they were significantly more successful at identifying the domain level. Researchers found that approximately 48.7% of the sites cited by the model could be identified via these methods.
This discrepancy is vital for SEO professionals. It suggests that while we can predict which domains might gain visibility in an AI response, the specific pages are chosen through a logic gate that standard search algorithms do not fully replicate.
The Variability of AI Behavior
Perhaps the most troubling finding for those relying on ChatGPT for consistent information is the "run-to-run" variation. For GPT-5.5 Instant, 44.5% of the pages cited for a specific question appeared in only one of the 20 iterations. This indicates that citation selection is not merely a product of the highest-ranked search result, but is instead influenced by the "state" of the model, prompt transformation, and potentially randomized weightings in the model’s retrieval-augmented generation (RAG) pipeline.
Language and Query Manipulation
The research confirms that ChatGPT does not operate as a passive search proxy. In the vast majority of cases, the model rewrote the user’s original prompt, opting for its own internal search terminology. Furthermore, the model exhibited a pronounced linguistic bias. Even when prompts were submitted in Chinese, GPT-5.3 and GPT-5.5 Instant defaulted to English-language searches in 67.0% and 69.6% of runs, respectively. This aligns with previous industry reports suggesting that non-English queries often suffer from a "fan-out" effect, where the model pivots to English-language datasets to maximize the breadth of potential sources.
Official Statements and Industry Context
OpenAI has previously stated that when ChatGPT performs a web search, it transforms the user’s input into one or more search terms to improve retrieval efficacy. However, the researchers argue that this "black box" transformation renders external auditing tools nearly useless.
The researchers noted that their findings correlate with previous studies, such as the "CiteChoice" preprint, which found that rerunning identical inputs can flip "cite-or-not" decisions by as much as 15%. This inherent instability means that any single-snapshot analysis of an AI model’s citation patterns is inherently incomplete.
In February, independent analysis of "Peec AI" data indicated that 43% of non-English prompts triggered English-language queries, a figure that the new research suggests may be even higher depending on the specific model version. As the industry moves toward more sophisticated iterations, these metrics serve as a baseline for the degree of "algorithmic drift" that users should expect.
Future Outlook: Moving Toward GPT-6
The findings of this paper represent a snapshot in time—specifically the landscape of AI search as it existed through the summer of 2026. Since the conclusion of the data collection, OpenAI has begun rolling out GPT-6 and the GPT-5.6 "Luna" model.
The Need for Dynamic Monitoring
The authors of the study suggest that traditional, static monitoring of AI citations is no longer viable. Instead, they propose a multi-layered approach:
- Multi-Run Benchmarking: Because of the high variability between runs, monitoring must involve repeated queries to establish a confidence interval for which sites are consistently cited.
- Domain-Level Focus: Given that site-level matching is significantly more successful than page-level matching, analysts should prioritize domain-authority tracking over individual URL indexing.
- Awareness of Model Drift: Every update to the underlying model—such as the transition from GPT-5.5 to GPT-5.6—requires a recalibration of the monitoring strategy. As evidenced by Sistrix data from earlier in the year, citation behavior can fluctuate wildly (up to 47% variance in some languages) when a model version is updated.
A New Era of Transparency?
While the preprint has not undergone formal peer review, it provides a crucial framework for understanding the future of digital discovery. As ChatGPT and its competitors become the primary interface through which users consume information, the "source selection" process will become the most important battleground in the search industry.
For the time being, the research serves as a sobering reminder: when you ask ChatGPT for a source, you are not receiving an objective list of the "best" results. You are receiving a probabilistic, highly variable output influenced by internal linguistic biases, prompt-transformation logic, and the specific version of the model running at that exact micro-second.
As we look toward the widespread adoption of GPT-6, the challenge for the research community and for SEOs will be to develop tools that can peer inside these dynamic systems, not just to track where we are being cited, but to understand why the machine chooses one voice over another. Until then, the "black box" remains, and the gap between a search query and a citation remains a frontier of unpredictable algorithmic design.
