Large language model (LLM)-assisted literature retrieval may lead to bibliographic errors that have not been rigorously quantified. Therefore, we aimed to analyze errors in reference retrieval of free-version LLMs and their associated factors.
We evaluated 2,000 references retrieved by 5 LLMs (Grok, ChatGPT, Gemini, Perplexity, and DeepSeek) for 40 randomly selected original articles (10 per journal) published from January 2024 to July 2025 in British Medical Journal, Journal of the American Medical Association, Lancet, and New England Journal of Medicine. The primary outcomes were the score ratio, a composite score of validity of digital object identifier, PubMed ID, Google Scholar link, and relevance, with a bonus point for retrieving the index article, and the complete miss status that indicated failures on all metrics. Multilevel mixed-effects multivariable regression was used.
References retrieved by LLMs failed all metrics 48.7% of the time. Grok was linked to a higher score ratio than ChatGPT (coefficient = 0.14, 95% confidence interval [CI], 0.07–0.21; P < 0.001), while all other platforms were associated with lower ones. Compared with ChatGPT, Grok was associated with lower odds of complete miss (better performance, odds ratio (OR) = 0.23, 95% CI, 0.11–0.46; P < 0.001), but DeepSeek, Gemini, and Perplexity were associated with higher odds (OR = 3.49, 95% CI, 1.87–6.50; OR = 16.86, 95% CI, 8.69–32.71; OR = 11.01, 95% CI, 5.79–20.94; respectively; P < 0.001 for all). Journals were linked to the score ratio, not complete miss.
We show that the performance of LLM-assisted literature search and retrieval in medicine is overall variable and often unreliable.
Full article