Claude AI and Literature Reviews: An Experiment in Utility and Ethical Use
Generative large-language models (LLMs) like ChatGPT and Claude have sparked a vibrant debate in higher education, particularly when it comes to traditional paper-based research and assignments. In addition to undisclosed use of generative AI making it difficult (or even impossible) to gauge a student’s understanding of a topic, it also has the potential to compromise the quality and integrity of scholarly research due to hallucinations from the AI. But will generative AI models continue to hallucinate when given the full text of an article to “read”?
Fellow librarian Alan Witt and I decided to do a deep dive into the literature surrounding how to use (or how not to use) generative AI in research and instruction, as well as finding out where exactly a generative AI model’s (in this case, Anthropic’s Claude) strengths and weaknesses in writing are. Once we gathered our collection of sources, I went about writing a standard thematic literature review highlighting the benefits and drawbacks of using generative AI in research and pedagogy. After, I fed the PDFs of the articles I used into Claude (with the original author’s permission) and asked it to write a literature review on the same topic with the same sources. Then, Alan went in and did a comparison of the two literature reviews, including identifying the strengths and weaknesses in both.
In theory, the AI would hallucinate a lot less since it had the original text of the article to draw from. In practice, this appeared to be the case on a broad level, but upon closer inspection there were still plenty of errors, including incorrect page numbers and quotes taken out of context to argue the opposite point the original author was trying to make. So while the generative AI model was able to read through the articles and pull all the pertinent information out, it was less successful in actually creating a well-supported argument from the information it gathered.
Claude also struggled to provide any deep analysis or synthesis of the sources, instead leaving quotations to speak for themselves as individual pieces of information rather than as part of a larger picture. That said, Claude excelled at summarizing the information and providing a cursory list of connections between sources.
At the end of the day, there are definitely potential uses for generative AI in writing, but significant human intervention is required before anything the model produces is actually usable. We also found trying to go about creating this literature review while respecting intellectual property to be incredibly time-consuming. This was amplified by the fact that we used the free version of Claude, which limited the number of PDFs we could upload in a single batch and the number of responses we could get per day.
The goal of our research was to fill an existing gap in the literature, as most studies we found revolved around the use of ChatGPT and did very little exploration of other models. Most studies we found also did not provide the model with the original sources, which could be setting the model up for failure. Our intent was to see how well an AI-generated literature review stands up to a human-created literature review, especially when the AI has access to the same (or mostly the same) resources as the human.
“Image Credit: Claude Rains; ''Deception'', 1946” by Movie-Fan is licensed under CC BY-NC-SA 2.0 
.