Executive Overview: The Transparency Gap
As Artificial Intelligence models become the primary gateway for information retrieval, the mechanism by which they select and cite sources has become a subject of intense scrutiny. A groundbreaking preprint paper, released on October 11, 2026, by researchers from the National University of Singapore and Xi’an Jiaotong University, sheds light on the opaque "citation behavior" of OpenAI’s ChatGPT.
The study, which analyzed models ranging from GPT-5.2 Instant to GPT-5.5 Instant, reveals a startling disconnect between the information accessible to the public via search engines and the specific content ChatGPT chooses to prioritize. The research found that when comparing ChatGPT’s web-browsing output against a combination of Google search results and OpenAI’s own Responses API, there is a significant failure in overlapping citations. Specifically, for the GPT-5.5 Instant model, the combined external data sources identified only 26.2% of the specific pages cited by the chatbot. While the model showed higher success in identifying the domain (finding 48.7% of the sites behind those citations), the inability to predict or replicate specific page-level citations highlights a fundamental instability in how AI platforms synthesize information.
This investigative study underscores a critical reality for developers, SEO professionals, and casual users alike: the "sources" provided by AI are not necessarily the most relevant or popular content available on the open web, but rather the result of a highly volatile, internal transformation process that remains largely shielded from external audit.
Detailed Chronology: Tracking the AI’s Decision Flow
To understand how ChatGPT arrives at its citations, researchers conducted a longitudinal study spanning four months, from March 29 to July 29, 2026. The methodology was designed to bypass the "black box" nature of the ChatGPT web interface by recording the raw data flow within active, signed-in user sessions.
The Methodology of Observation
The research team tracked 80 distinct prompts sourced from Search Arena, a public database of real-world user interactions with search-enabled AI models. These prompts were categorized into four functional types: fact lookup, information synthesis, analysis, and recommendations. To ensure linguistic robustness, the test set was split evenly between 40 English and 40 Chinese queries.
Each query was executed 20 times across different models, including GPT-5.3 Instant and GPT-5.5 Instant, to account for the stochastic nature of Large Language Models (LLMs). The researchers captured the entire "thought" process: the initial query, the subsequent search terms generated by the AI, the pages retrieved, and the final selection of cited sources.
The Evolution of Models
The study captured a snapshot of OpenAI’s rapid iteration cycle. GPT-5.2 Instant and GPT-5.4 Thinking were decommissioned mid-study, necessitating a focus on the 5.3 and 5.5 versions for the majority of the comparative analysis. This rapid deprecation of models highlights a significant hurdle for researchers: by the time an analysis is completed, the underlying architecture of the AI has often shifted, rendering previous findings obsolete.
Supporting Context & Metrics: The Discrepancy Analysis
The core of the study lies in the comparison between ChatGPT’s cited pages and the results retrieved through standard search vectors (Google Organic and OpenAI’s Responses API).
Page vs. Site Attribution
The researchers found that while site-level matching was relatively successful—reaching over 50% for GPT-5.3—page-level matching remained remarkably low. This implies that while ChatGPT might correctly identify a reputable publisher (like a major news site or an official documentation hub), it frequently chooses different specific URLs than those returned by traditional search algorithms.
- GPT-5.3 Instant: Cited pages matched 28.3% of API/Google results.
- GPT-5.5 Instant: Cited pages matched 26.2% of API/Google results.
These figures suggest that ChatGPT’s internal weighting for "source worthiness" deviates significantly from the ranking signals used by traditional search engines. The API, while providing a closer match than raw Google search results, still fails to account for the majority of the chatbot’s selections, suggesting that ChatGPT applies a proprietary "re-ranking" or "filtering" layer that is not reflected in standard search API outputs.
The Problem of Internal Query Rewriting
A key finding in the paper is that ChatGPT almost never uses the user’s original prompt as its search term. OpenAI’s documentation confirms that the model transforms queries into internal search terms to better suit its retrieval infrastructure. However, the study shows this process is prone to significant "hallucination of intent."
In the case of Chinese-language queries, the model exhibited a strong bias toward English-language search terms. In tests involving GPT-5.3 and 5.5, roughly 70% of searches triggered by Chinese prompts were performed entirely in English. This creates a "translation bottleneck" where the information retrieved is limited to the English-speaking web, potentially ignoring high-quality, native-language sources that would have been more relevant to the original user query.
The Instability of "Truth"
Perhaps the most alarming metric for those relying on AI for research is the variability of output between identical runs. The study found that nearly 45% of the distinct pages cited for a single question appeared in only one of the 20 experimental runs.
This indicates that the "source list" provided by ChatGPT is not a stable or curated bibliography, but a snapshot of a highly non-deterministic process. Even when presented with the exact same input, the AI may cite entirely different sources in a subsequent session. For professionals, this confirms that citing AI-generated content remains a high-risk activity, as the underlying evidence base is subject to change without notice.
Official Statements and Industry Implications
OpenAI has consistently maintained that its search-enabled models are designed to provide synthesis rather than a simple search-engine-style list of links. However, the researchers argue that the lack of consistency in citation makes it difficult for stakeholders—particularly content creators and SEOs—to understand how their work is being surfaced (or buried) by the model.
Monitoring Strategies
The authors of the paper propose a three-pronged approach for those looking to track AI visibility:
- Multi-Model Validation: Do not rely on a single model version; as ChatGPT updates, the underlying logic shifts significantly.
- Site-Level Aggregation: Focus on domain-level presence rather than specific page URLs, as the model’s internal link selection is highly volatile.
- Prompt Sensitivity: Acknowledge that the AI is rewriting every prompt. Future monitoring tools must account for the "translation layer" between the user’s input and the model’s final search query.
The paper notably stops short of endorsing any commercial tracking tools, suggesting that existing market solutions are not yet sophisticated enough to mirror the complex, internal decision-making process observed in the study.
Future Outlook: The Era of GPT-6
The release of this preprint coincides with the rollout of GPT-6 and GPT-5.6 Luna. As these models become the standard, the findings regarding GPT-5.5 will inevitably face a new cycle of re-evaluation.
The volatility noted in this study—where citation patterns fluctuate wildly—is likely to be a defining feature of the "Agentic AI" era. As AI models move toward acting as autonomous agents, they will increasingly prioritize "actionable data" over traditional relevance, potentially pushing traditional SEO metrics further into the background.
For the academic and research communities, the challenge remains clear: if we cannot determine why a model cites a specific page today, we cannot trust that it will provide a balanced or accurate set of information tomorrow. The research from the National University of Singapore and Xi’an Jiaotong University serves as a necessary wake-up call. It highlights that the "intelligence" in artificial intelligence often manifests as a chaotic, moving target, necessitating not just better algorithms, but a fundamental shift in how we monitor, audit, and trust the digital intermediaries that define our information landscape.
Until the scientific community can establish a standardized framework for auditing AI retrieval, users should view ChatGPT’s citations not as authoritative sources, but as dynamic, transient suggestions—subject to the unpredictable whims of a model that is constantly learning, forgetting, and rewriting the rules of the web.
