Executive Overview
The mechanics of how conversational artificial intelligence selects, surfaces, and cites web sources have long remained a digital black box for search engine optimization (SEO) professionals, publishers, and digital marketers. A groundbreaking preprint study released by researchers at the National University of Singapore and Xi’an Jiaotong University sheds unprecedented light on this opaque ecosystem.
Analyzing the internal data flows of OpenAI’s ChatGPT web application across multiple iterations—ranging from GPT-5.2 Instant to GPT-5.5 Instant—the researchers discovered a startling discrepancy between what AI search tools cite and what traditional web visibility metrics predict. According to the findings, combining Google search results and OpenAI’s own Responses API accounts for a mere 26.2% of the exact web pages cited by ChatGPT in its GPT-5.5 Instant setting. However, when evaluated at the broader domain level rather than individual URLs, that same combination manages to capture roughly 48.7% of cited sites.
This empirical disconnect underscores a fundamental challenge for the modern digital landscape: AI-driven search engines do not operate like traditional search engines. They dynamically rewrite user prompts, heavily favor English-language queries even when prompted in other languages, and exhibit immense variance between identical runs. As the industry transitions toward newer architectures like GPT-5.6 Luna and GPT-6, this landmark study provides critical methodological frameworks for anyone attempting to monitor, audit, or optimize brand visibility within generative search environments.
Detailed Chronology and Methodology: How Researchers Watched ChatGPT
To decode the mechanics of ChatGPT’s citation patterns, the research team implemented a rigorous, multi-layered data collection framework spanning several months. The primary window of data acquisition ran continuously from March 29 through July 29, 2026.
Capturing the Data Flow
Rather than merely observing surface-level output, the researchers monitored the live data streams received by ChatGPT’s web application during active writing sessions. By recording authenticated user sessions, the team gained granular visibility into:
- The exact underlying search queries executed by ChatGPT.
- The raw search engine result pages (SERPs) generated by those internal queries.
- The specific subset of pages chosen by the model to serve as authoritative citations.
For the purposes of the study, a page was formally designated as "cited" if ChatGPT explicitly selected it as a source—encompassing both individual hyperlinked URLs and grouped citation bundles treated as a single selection.
The Test Bed: Questions and Models
The study evaluated an array of 80 carefully curated questions extracted from Search Arena, a public repository documenting genuine user interactions with search-enabled AI models and chatbots. To ensure linguistic and functional diversity, the test bed comprised:
- 40 English queries and 40 Chinese queries.
- Four distinct prompt types: recommendations, fact lookups, information synthesis, and analytical requests.
Each of the 80 questions was executed 20 consecutive times across multiple model iterations, including GPT-5.3 Instant and GPT-5.5 Instant. Earlier iterations, specifically GPT-5.2 Instant and GPT-5.4 Thinking, were subjected to fewer test runs due to being decommissioned midway through the data collection cycle.
Establishing the Control Group (Stand-Ins)
To evaluate whether external tools could replicate ChatGPT’s web application behavior, the researchers constructed a dual-pronged control mechanism:
- Google Organic Search: Using SerpAPI, researchers retrieved up to five pages of Google organic results per question, averaging approximately 43 results per query.
- OpenAI Responses API: For every question, the team sent 20 parallel queries to
gpt-5.3-chat-latestandgpt-5.5via OpenAI’s Responses API with web search functionality enabled.
All control data was captured within a three-day window of the original ChatGPT web app runs to ensure temporal consistency. Because the paper was published as an arXiv preprint (arXiv:2610.15382) on October 11, 2026, it represents a non-peer-reviewed yet exceptionally thorough empirical snapshot of generative search behavior.
Supporting Context and Metrics: Stand-Ins Match Sites, Miss Pages
The core finding of the Singapore and Xi’an Jiaotong University study centers on the systemic gap between page-level visibility and domain-level visibility.
When combining Google’s search results and the OpenAI Responses API, the metrics reveal a stark division in predictive accuracy:
- GPT-5.3 Instant: The combination successfully surfaced 28.3% of the exact pages cited by the model, but successfully identified 50.3% of all cited domains (treating subdomains as distinct sites).
- GPT-5.5 Instant: The combination captured 26.2% of cited pages and 48.7% of cited sites.
Dissecting the Sources
Evaluating the components individually highlights further nuances in how generative models source information compared to standard APIs:
- Google Search Alone: Accounted for just 9.4% of cited pages in GPT-5.3 and 6.4% in GPT-5.5. However, its performance was significantly stronger at the site level, accounting for 26.1% and 22.8% of cited domains, respectively.
- OpenAI Responses API: The API’s complete source inventory performed better at the page level than Google alone, capturing 22.9% (GPT-5.3) and 22.5% (GPT-5.5) of all cited pages.
Interestingly, while the API’s own direct answer citations covered fewer pages overall, they proved to be a closer structural match to ChatGPT’s choices. Averaged across the 80 test questions, GPT-5.5 Instant cited 23.3% of the exact pages generated by the API in its own answers—a notable jump compared to the 16.4% overlap observed with GPT-5.3 Instant.
Behavioral Anomalies: Query Rewriting, Linguistic Bias, and Variance
Beyond raw matching percentages, the study uncovered critical behavioral quirks in how ChatGPT constructs searches and handles output consistency across multiple executions.
Query Transformation and Translation Bias
OpenAI’s official documentation notes that when ChatGPT utilizes web search via external providers, it routinely transforms raw user prompts into targeted, optimized search terms. The research paper confirms this phenomenon while revealing two major operational tendencies:
- Zero Exact-Wording Matches: In observed runs for GPT-5.3 Instant and GPT-5.5 Instant, ChatGPT never queried search providers using the exact wording of the original user prompt. (By contrast, the older GPT-5.2 Instant preserved exact wording in 5.5% of runs).
- The English-Language Hegemony: When processing non-English inputs, ChatGPT displayed a pronounced bias toward English search terms. Among visible searches originating from Chinese-language questions, 67.0% of GPT-5.3 Instant runs and 69.6% of GPT-5.5 Instant runs executed searches exclusively in English.
This reinforces findings from earlier industry analyses, such as Peec AI’s February telemetry data, which reported that 43% of fan-out queries for non-English prompts default to English execution.
High Inter-Run Variance
Marketers tracking AI visibility must also contend with extreme volatility between individual sessions. The study found that:
- 44.5% of distinct pages returned for a given query appeared in only one of the 20 GPT-5.5 Instant runs.
- 40.8% of distinct pages returned for a given query appeared in only one of the 20 GPT-5.3 Instant runs.
While site-level overlap proved more stable than page-level overlap, the sheer volume of single-run appearances complicates traditional rank-tracking methodologies. Furthermore, the researchers noted that while pages appearing frequently across multiple runs had a higher likelihood of being referenced, high frequency of appearance did not mathematically dictate citation selection. This aligns with earlier findings from the CiteChoice preprint, which demonstrated that rerunning identical inputs can flip roughly 15% of all cite-or-not decisions.
Official Industry Implications and Best Practices
The methodological divide highlighted by the Singapore and Xi’an Jiaotong University researchers carries profound implications for digital strategists, enterprise brands, and developers of commercial tracking tools.
Why Standard Tracking Falls Short
Because neither Google search rankings nor OpenAI’s API outputs mirror ChatGPT’s cited pages on a 1:1 basis, relying on conventional rank-tracking setups guarantees blind spots. Tools that monitor visibility exclusively through API endpoints or standard SERP scraping will consistently over-report site-level presence while missing the granular URL-level dynamics that dictate referral traffic.
Recommended Monitoring Strategies
Based on their empirical findings, the study authors advise auditing generative search visibility through a multi-faceted approach:
- Monitor at Dual Levels: Track brand visibility simultaneously at the individual webpage (URL) level and the broader domain level.
- Embrace Multi-Run Sampling: Never rely on a single API query or chat session. Because inter-run variance can exceed 40% for distinct pages, automated tracking systems must execute multiple parallel iterations of identical prompts.
- Account for Prompt Translation: Optimize content not just for the literal phrasing of customer queries, but for the semantic variations and English-centric search strings that conversational models generate under the hood.
- Identify the Underlying Model: Always verify whether comparison data is collected directly through the consumer ChatGPT web application or via backend developer APIs, as response behaviors diverge.
Future Outlook: The Horizon Beyond GPT-5.5
It is important to contextualize these findings within the rapid pace of artificial intelligence development. Every model tested in the preprint—spanning GPT-5.2 through GPT-5.5 Instant—predates the commercial rollouts of GPT-5.6 Luna and GPT-6.
As of late 2026, OpenAI’s deployment roadmap has shifted significantly: GPT-6 is actively rolling out to eligible Plus, Pro, Business, and Enterprise users, while GPT-5.6 Luna has established itself as the default engine for Free and Go tier accounts.
Historical data indicates that these architectural shifts trigger massive volatility in citation dynamics. For instance, empirical telemetry from Sistrix surrounding the GPT-5.5 rollout in May demonstrated that day-to-day citation variations for German-language queries spiked to 47%, obliterating the historical baseline of 1% to 2% daily fluctuation.
The study’s authors emphasize that monitoring frameworks must be continuously re-evaluated and recalibrated each time OpenAI updates the underlying foundational model. Until independent academic teams replicate these comprehensive tests on GPT-6 and its successors, the findings from the National University of Singapore and Xi’an Jiaotong University serve as the definitive baseline—illuminating the complex, unpredictable, and highly dynamic nature of AI-generated citations.
