EXECUTIVE OVERVIEW
For digital marketers, search engine optimization (SEO) professionals, and content creators, understanding how artificial intelligence cites web sources has become a high-stakes guessing game. A compelling new preprint paper released by researchers at the National University of Singapore and Xi’an Jiaotong University offers a rare, granular look under the hood of OpenAI’s web-enabled architecture.
The study reveals a staggering disconnect: when comparing standard search algorithms and API-driven data against the live user experience of ChatGPT, traditional monitoring tools fail to capture the vast majority of exact webpage citations. Specifically, Google search results and OpenAI’s own Responses API together managed to uncover only 26.2% of the precise pages cited by ChatGPT in tests of the GPT-5.5 Instant setting. However, when evaluated at the broader domain level, these tracking methods performed significantly better, identifying nearly half—48.7%—of the underlying websites behind those citations.
Published on arXiv on October 11, the paper—titled Tracing AI Citations: A Comparative Analysis of Web-Enabled LLMs—sheds light on the volatile, heavily abstracted mechanisms powering modern conversational search. By tracking live data flows within signed-in user sessions across multiple iterations of the GPT-5 model family (ranging from GPT-5.2 Instant to GPT-5.5 Instant), the researchers exposed systemic behaviors that challenge conventional wisdom regarding how generative engines discover, process, and attribute web content.
As the industry pivots toward AI-driven discovery, these findings carry profound implications for brand visibility, referral traffic tracking, and the future of digital measurement. This report breaks down the methodology, key metrics, and strategic takeaways from the Singapore-Xi’an Jiaotong study, offering an exhaustive look at why tracking AI citations is far more complex than previously imagined.
METHODOLOGY AND SCOPE: HOW RESEARCHERS WATCHED CHATGPT
To pierce the veil of OpenAI’s proprietary ecosystem, the research team adopted an innovative observational approach. Rather than relying solely on external API queries, they monitored the live data streams transmitted to and from ChatGPT’s web application during active, signed-in user sessions.
This methodology allowed researchers to intercept the internal lifecycle of an AI query:
- The Query Transformation: Observing how user prompts were dynamically rewritten by ChatGPT into specialized search strings.
- The Discovery Phase: Documenting the specific search engine results pages (SERPs) triggered by those translated queries.
- The Selection Phase: Recording which exact web pages ChatGPT ultimately chose to ingest, synthesize, and cite as authoritative sources.
The Test Dataset: Search Arena and Multilingual Inputs
The study utilized 80 carefully curated questions drawn from Search Arena, a public database cataloging real-world user interactions with search-enabled AI models. To test linguistic biases and handling capabilities, the dataset was split evenly:
- 40 English-language prompts
- 40 Chinese-language prompts
Furthermore, the questions spanned four distinct intent categories:
- Recommendation queries
- Fact-lookup queries
- Information synthesis tasks
- Complex analytical prompts
To establish statistical significance, each of the 80 questions was executed 20 distinct times across multiple model iterations—primarily focusing on GPT-5.3 Instant and GPT-5.5 Instant. (While older iterations like GPT-5.2 Instant and GPT-5.4 Thinking were included in early phases, they were decommissioned prior to data collection completion and yielded smaller sample sizes).
Replicating the Environment via APIs and SerpAPI
To benchmark these results against external proxies, the researchers attempted to recreate ChatGPT’s search environment using two external tools:
- SerpAPI: Used to retrieve up to five pages of Google organic search results per question (averaging roughly 43 results per query).
- OpenAI’s Responses API: Sent 20 queries per question using
gpt-5.3-chat-latestandgpt-5.5endpoints with web search enabled.
All external API queries and SERP extractions were performed within a strict three-day window relative to the live ChatGPT runs to ensure temporal consistency. Because the study was published as an arXiv preprint, it has not yet undergone formal peer review, yet its empirical observations align closely with emerging frustrations across the SEO community.
SUPPORTING CONTEXT & METRICS: STAND-INS MATCH SITES MORE OFTEN THAN PAGES
The core finding of the Singapore-Xi’an Jiaotong study centers on a glaring discrepancy: external monitoring tools are profoundly misaligned with the micro-citations generated by live chat interfaces.
When combining Google search results with OpenAI’s own Responses API data, the overlap with actual ChatGPT citations was remarkably low at the individual page level:
- GPT-5.3 Instant: Combined tools found 28.3% of cited pages.
- GPT-5.5 Instant: Combined tools found 26.2% of cited pages.
However, when evaluated at the macro domain level (treating subdomains as distinct sites), the visibility metrics improved dramatically:
- GPT-5.3 Instant: Combined tools found 50.3% of cited sites.
- GPT-5.5 Instant: Combined tools found 48.7% of cited sites.
Dissecting the Tools: Google vs. API Performance
Breaking down the data reveals distinct performance gaps between traditional search engines and native API environments:
- Google Search Alone: Accounted for just 9.4% (GPT-5.3) and 6.4% (GPT-5.5) of cited pages. At the site level, Google performed stronger, accounting for 26.1% and 22.8% of cited domains, respectively.
- OpenAI Responses API: The API’s entire source list captured 22.9% (GPT-5.3) and 22.5% (GPT-5.5) of cited pages—outperforming Google’s organic results. Furthermore, when comparing API-generated answers directly to ChatGPT’s app-based citations, the API demonstrated a closer structural match, with GPT-5.5 Instant citing 23.3% of the pages generated by the API (compared to just 16.4% for GPT-5.3 Instant).
The Phenomenon of Prompt Rewriting and Linguistic Shift
A major reason external tools struggle to replicate AI citations lies in how ChatGPT interacts with user input. According to official OpenAI developer guidelines, when web search is invoked, ChatGPT rarely uses a user’s prompt verbatim. Instead, it acts as an intelligent intermediary, transforming a single natural language question into one or more sophisticated search strings.
The preprint data confirms this behavior on a mass scale:
- In tests involving GPT-5.3 Instant and GPT-5.5 Instant, ChatGPT never once queried using the exact wording of the original user prompt. (By contrast, the legacy GPT-5.2 Instant used exact wording in a meager 5.5% of runs).
- The Language Translation Bias: Perhaps most striking for international brands is the model’s tendency to force non-English queries into English. Among visible searches stemming from Chinese-language prompts, GPT-5.3 Instant utilized English in 67.0% of runs, while GPT-5.5 Instant pushed that figure to 69.6%.
This mirrors broader industry observations. Earlier findings from Peec AI highlighted that roughly 43% of "fan-out" queries for non-English prompts default to English, severely impacting local search visibility for regional publishers.
Volatility Between Runs: The Illusion of Consistency
Marketers tracking AI visibility often panic or celebrate based on a single snapshot. The preprint demonstrates that this is a statistical trap.
When researchers re-ran identical inputs, the results fluctuated wildly:
- 44.5% of distinct pages returned for a specific question appeared in only one of the 20 GPT-5.5 Instant runs.
- 40.8% of distinct pages for GPT-5.3 Instant showed the exact same single-run volatility.
While domain-level overlap remained slightly more stable across runs, the sheer volume of transient citations underscores how fluid generative search engines are. This reinforces previous findings from the CiteChoice preprint, which demonstrated that minor fluctuations in runtime parameters can flip 15% of cite-or-not attribution decisions.
OFFICIAL STATEMENTS AND TECHNICAL IMPLICATIONS
The implications of these findings force a fundamental rethink of how digital marketing analytics must evolve. Because neither Google’s ranking algorithms nor OpenAI’s developer APIs mirror the exact webpage output of the consumer-facing ChatGPT web application, relying on standard ranking tools provides a distorted view of brand visibility.
The study’s authors put forward several practical recommendations for enterprise monitoring frameworks:
- Adopt Dual-Level Tracking: Brands must monitor their footprint at both the individual webpage level and the broader site/domain level.
- Account for Multi-Run Variance: Because single-query testing is inherently unreliable, visibility audits must execute prompts multiple times across different sessions to establish a probabilistic baseline.
- Decode Prompt Abstraction: Monitoring tools must account for query rewriting. Analysts should actively check whether tracking software evaluates raw user inputs or the transformed secondary search strings actually dispatched to the web.
- Distinguish Access Vectors: Digital strategists must differentiate between data collected via developer APIs versus telemetry captured directly from consumer applications, as the citation distributions diverge significantly.
FUTURE OUTLOOK: THE ROAD TO GPT-6 AND BEYOND
As is often the case with rapid advancements in generative artificial intelligence, temporal obsolescence is a constant hurdle. The models analyzed in the Singapore-Xi’an Jiaotong preprint—spanning iterations up to GPT-5.5 Instant—predate the current architectural rollouts of GPT-5.6 Luna and the enterprise-wide deployment of GPT-6, which is actively rolling out to ChatGPT Plus, Pro, Business, and Enterprise subscribers.
Historically, major model upgrades dramatically reshape citation dynamics. For instance, data from Sistrix surrounding the transition to GPT-5.5 showed day-to-day citation volatility in German-language queries skyrocketing to 47%, a massive leap from the historical baseline of 1% to 2%.
As OpenAI continues to refine its search grounding, retrieval-augmented generation (RAG) pipelines, and real-time synthesis layers, industry experts anticipate that citation stability will remain elusive. Researchers urge the analytics community to continuously re-evaluate monitoring strategies every time an underlying model architecture shifts.
Until comprehensive replication studies are completed on GPT-6 and its successors, the findings from the National University of Singapore and Xi’an Jiaotong University serve as both a cautionary tale and an operational roadmap. For SEOs and digital strategists, the message is clear: the era of simplistic keyword rankings is dead. Navigating the AI-driven web requires understanding the probabilistic, shifting black box of generative citation.
