Executive Overview
As generative search increasingly supplants traditional web traffic channels, digital marketers, SEO professionals, and content creators face a persistent, high-stakes puzzle: How does an AI model actually choose its sources, and how can we track them? A comprehensive new preprint paper released by researchers at the National University of Singapore and Xi’an Jiaotong University offers a rare, microscopic look under the hood of ChatGPT’s citation engine.
The study—which analyzed data collected between March 29 and July 29, 2026—reveals a startling disconnect between the websites standard SEO monitoring tools capture and the actual web pages cited by ChatGPT. According to the findings, when researchers utilized Google search results and OpenAI’s own Responses API as proxies to replicate ChatGPT’s web-fetching behavior, the combination managed to capture only about 26.2% of the specific pages cited by the GPT-5.5 Instant setting. However, when evaluated at the broader domain level, that same combination captured nearly half—48.7%—of all cited sites.
These findings underscore a fundamental challenge for the emerging discipline of Generative Engine Optimization (GEO): ChatGPT does not process queries, perform searches, or surface citations the way a traditional search engine does. By rewriting user inputs, heavily favoring English for non-English queries, and generating immense variance between consecutive runs, OpenAI’s models treat citations as a dynamic, probabilistic output rather than a static directory of links. As the industry transitions toward newer architectures like GPT-5.6 and GPT-6, this landmark study serves as both a cautionary tale for current tracking methodologies and a blueprint for how researchers and marketers must adapt their monitoring strategies.
Detailed Chronology and Methodology: How Researchers Watched ChatGPT
To understand how ChatGPT arrives at its web citations, the research team—whose paper was published to arXiv on October 11—had to design a rigorous, multi-layered data collection framework capable of capturing the underlying mechanics of OpenAI’s web application.
Capturing the Data Flow
Rather than simply prompting the public-facing chat interface and recording the final output, the researchers monitored the live data streams received by ChatGPT’s web application during active, signed-in user sessions. This telemetry data flow exposed the invisible gears of the AI’s research phase: the exact search queries formulated by ChatGPT, the raw organic search pages returned by those queries, and the specific web properties ultimately selected as authoritative sources.
Within this methodology, a web page was classified as "cited" if ChatGPT explicitly selected it as a reference source. This included both standalone URLs and grouped selections bundled together under a single citation marker.
The Test Suite: Prompts and Models
The dataset was built upon 80 distinct questions sourced from Search Arena, a public repository tracking actual user interactions with search-enabled conversational models. To ensure linguistic and functional diversity, the test suite comprised:
- Languages: 40 English questions and 40 Chinese questions.
- Intent Types: Four distinct categories of user requests—recommendations, fact lookups, information synthesis, and complex analysis.
The researchers tested four successive iterations of OpenAI’s architecture: GPT-5.2 Instant, GPT-5.3 Instant, GPT-5.4 Thinking, and GPT-5.5 Instant. Because GPT-5.2 Instant and GPT-5.4 Thinking were decommissioned mid-study, the bulk of the comparative data collection concentrated on GPT-5.3 Instant and GPT-5.5 Instant. Each of the 80 test questions was executed 20 times across these active models to measure output stability.
The Proxy Setup: Google Search and the Responses API
To evaluate whether external developer tools could accurately mimic ChatGPT’s internal research behavior, the researchers established two benchmark proxies:
- Google Organic Results: Utilizing SerpAPI, the researchers retrieved up to five pages of organic Google results per question, averaging roughly 43 indexed URLs per prompt.
- OpenAI Responses API: For every question, 20 queries were routed through the API using the
gpt-5.3-chat-latestandgpt-5.5endpoints with web search enabled.
All proxy data collection was executed within a strict three-day window of the original ChatGPT web app runs to eliminate temporal drift in search engine indexes.
Supporting Context & Metrics: Stand-Ins Match Sites, Not Pages
The core finding of the Singapore-Xi’an Jiaotong study centers on a stark discrepancy: proxy tools are relatively good at identifying which websites ChatGPT draws from, but remarkably poor at predicting which exact pages will be referenced.
The Page vs. Site Disparity
When combining Google search results and the Responses API, the proxy tools successfully surfaced:
- GPT-5.3 Instant: 28.3% of cited pages, but 50.3% of cited sites.
- GPT-5.5 Instant: 26.2% of cited pages, but 48.7% of cited sites.
(Note: Subdomains were treated as distinct sites in these calculations).
When evaluated in isolation, Google’s organic search results accounted for a minor fraction of the actual references—capturing just 9.4% of cited pages for GPT-5.3 and 6.4% for GPT-5.5. However, Google performed much better at the domain level, uncovering 26.1% and 22.8% of cited sites, respectively.
Meanwhile, OpenAI’s Responses API source lists accounted for 22.9% of cited pages for GPT-5.3 and 22.5% for GPT-5.5. While the API’s own generated answers matched ChatGPT’s final citation choices less frequently overall, its broader reference lists offered a closer thematic alignment with the chatbot’s internal research path. For instance, when averaged across the 80 questions, GPT-5.5 Instant cited 23.3% of the exact pages that the Responses API had independently cited—a noticeable increase over GPT-5.3 Instant, which matched only 16.4% of API-cited pages.
Query Transformation and the English-Language Bias
One of the most profound revelations of the study involves how ChatGPT alters user prompts before sending them to search providers. According to OpenAI’s official documentation, search-enabled ChatGPT typically translates a conversational prompt into one or more optimized search strings.
The data confirmed this behavior dramatically: across thousands of observed runs for GPT-5.3 Instant and GPT-5.5 Instant, ChatGPT never once queried a search engine using the exact wording of the original user prompt. (By contrast, the older GPT-5.2 Instant preserved verbatim phrasing in 5.5% of runs).
Furthermore, the study highlighted a distinct linguistic bias. Among runs featuring visible searches for Chinese-language prompts, GPT-5.3 Instant executed searches exclusively in English 67.0% of the time, while GPT-5.5 Instant did so in 69.6% of runs. This aligns with earlier industry findings—such as a February report by Peec AI, which noted that 43% of non-English fan-out queries executed in English. For international brands and localized SEO strategies, this systemic anglicization represents a critical vulnerability in visibility tracking.
Run-to-Run Volatility
Compounding the challenge for digital marketers is the immense volatility observed between identical prompt executions. The researchers discovered that:
- 44.5% of distinct pages returned for a given question appeared in only one of the 20 GPT-5.5 Instant runs.
- 40.8% of distinct pages returned for a given question appeared in only one of the 20 GPT-5.3 Instant runs.
While site-level overlap was slightly more stable than page-level overlap, the sheer amount of churn underscores that AI citations are not deterministic. This matches recent industry discussions regarding the CiteChoice preprint, which demonstrated that rerunning identical inputs can flip up to 15% of citation decisions.
Official Statements and Industry Implications
The implications of this research extend far beyond academic curiosity, striking at the heart of commercial GEO and AI monitoring software. Because commercial tracking tools typically rely on API queries or simulated search scrapers to estimate a brand’s share of voice in generative models, this study proves that such tools operate with an inherent structural blind spot.
As the paper’s authors emphasize, neither Google search results nor OpenAI’s API results capture ChatGPT’s cited pages exactly as they appear in the end-user application. Consequently, any tool measuring webpage-level visibility via proxy APIs will consistently underreport precise URL citations while overestimating broad domain presence.
To mitigate these tracking errors, the researchers recommend that analysts adopt a multi-tiered monitoring approach:
- Dual-Level Tracking: Monitor visibility at both the granular webpage level and the broader domain level, acknowledging that domain-level metrics will offer a more reliable baseline.
- Repeated Prompt Execution: Because single-run tests are prone to extreme volatility (with over 40% of pages appearing in only one of twenty iterations), tracking frameworks must average results across multiple query runs.
- Account for Query Rewriting: SEO professionals must account for the fact that AI models fundamentally rewrite user prompts and frequently translate non-English queries into English before executing searches.
- Distinguish Data Sources: Analysts must audit whether their tracking software collects data directly from user-facing chat applications or via developer-facing APIs, as the underlying citation logic differs between the two.
Future Outlook: The Horizon Beyond GPT-5.5
While this study provides an unprecedented diagnostic lens into OpenAI’s models, the rapid pace of artificial intelligence development means the baseline is already shifting.
Every model evaluated in the Singapore-Xi’an Jiaotong paper predates the rollouts of GPT-5.6 and GPT-6. OpenAI’s current deployment schedule sees GPT-6 rolling out to eligible Plus, Pro, Business, and Enterprise users, while GPT-5.6 Luna has been established as the default model for Free and Go tiers.
Historical data suggests that major model transitions introduce massive turbulence into citation ecosystems. For instance, around the deployment of the GPT-5.5 update in May, data from Sistrix revealed that day-to-day variation in German-language ChatGPT citations spiked to an astonishing 47%—a dramatic departure from the normal 1% to 2% baseline fluctuation.
The study’s authors stress that whenever the underlying model architecture changes, digital marketers must completely re-evaluate their monitoring strategies. Until these rigorous citation-matching experiments are replicated on GPT-6 and its successors, the findings from this preprint serve as a definitive baseline for the Spring–Summer 2026 model generation—and a clear warning to the SEO industry that tracking the AI black box requires looking far beyond simple keyword rankings.
