Evaluating AI-generated literature reviews : a comparative analysis of ChatGPT and Gemini | |
| Author | Labh, Awaneendra Kishor |
| Call Number | AIT Thesis no.IM-25-01 |
| Subject(s) | Generative artificial intelligence Academic writing--Data processing Academic writing--Evaluation ChatGPT |
| Note | A thesis submitted in partial fulfillment of the requirements for the degree of Master of Engineering in Information Management |
| Publisher | Asian Institute of Technology |
| Abstract | In this research, literature reviews generated using our ChatGPT and Gemini against human-authored reviews from three different academic domains were analyzed. Using a corpus of 180 articles, a mixed-method design that combines qualitative expert assessment (accuracy, clarity, completeness, and source links) with quantitative measures (ROUGE scores, Cosine Similarity, sentiment analysis, and feature extraction with RoBERTa) has been designed. Both unsupervised clustering and supervised machine learning models were used. Such results suggest that the semantically closest paraphrased outputs from ChatGPT to human-authored reviews closely match the original reviews, where Gemini performs moderately lower but still acceptable, and full AI-generated reviews also show evidence of underdeveloped critical reasoning, referencing, and domain-specific depth, especially in narrative-heavy domains. The research demonstrates the promise and moral peril of using generative AI for scholarly work, recommending a hybrid model that combines human and machine intelligence to maintain the rigor, trustworthiness, and ethical use of such technology. |
| Year | 2025 |
| Type | Thesis |
| School | School of Engineering and Technology |
| Department | Department of Information and Communications Technologies (DICT) |
| Academic Program/FoS | Information Management (IM) |
| Chairperson(s) | Chutiporn Anutariya |
| Examination Committee(s) | Chantri Polprasert;Aekavute Sujarae |
| Scholarship Donor(s) | AIT Scholarship |
| Degree | Thesis (M. Eng.) - Asian Institute of Technology, 2025 |