Evaluating AI-Generated Physics Solutions: A Systematic Review of Solution Accuracy, Student Evaluation, and Implications for Critical Thinking in Higher Physics Education
DOI:
https://doi.org/10.47945/create.v1i1.3355Keywords:
Generative AI, Large Language Models, Physics Education, Critical Thinking, Learning From Errors, AI Literacy, Systematic ReviewAbstract
Generative artificial intelligence (AI) is now capable of producing coherent, step-by-step solutions to physics problems. However, its implications for physics education are determined more by students’ ability to evaluate the validity of these solutions than by their availability. This systematic review, reported with reference to the PRISMA 2020 guidelines, synthesizes peer-reviewed physics education studies published between 2020 and 2026 to address three questions: (1) how accurately does generative AI solve or explain physics tasks, and what types of errors does it produce; (2) how do students evaluate AI-generated physics responses; and (3) what instructional opportunities and risks emerge for the development of critical thinking in higher physics education? Eleven studies were synthesized through deductive and inductive thematic analysis. AI accuracy was highly dependent on task type. Large language models performed at levels comparable to or above those of university students on several concept inventories and olympiad problems, but solved only 8.3% of underspecified problems (compared with 62.5% of well-specified problems), misinterpreted graphs and visual representations, and generated contradictory explanations. The dominant errors were related to physical modeling, assumptions, and representations rather than arithmetic. Students often failed to detect these errors: first- and second-year physics students (n = 102) rated incorrect AI responses to the most difficult problems as equivalent to correct worked solutions. Instructional studies reported positive attitudes and beneficial learning support, but none measured improvements in critical thinking using validated instruments. Thus, the claim that evaluating AI solutions develops critical thinking is theoretically plausible, grounded in research on learning from errors and critical-thinking skills, but has not yet been empirically demonstrated in physics education. This review proposes an Evaluate–Diagnose–Correct–Reflect framework and a research agenda that distinguishes procedural correctness from conceptual validity.
References
Bastani, H., Bastani, O., Sungu, A., Ge, H., Kabakcı, Ö., & Mariman, R. (2025). Generative AI without guardrails can harm learning: Evidence from high school mathematics. Proceedings of the National Academy of Sciences, 122(26), Article e2422633122. https://doi.org/10.1073/pnas.2422633122
Bitzenbauer, P. (2023). ChatGPT in physics education: A pilot study on easy-to-implement activities. Contemporary Educational Technology, 15(3), Article ep430. https://doi.org/10.30935/cedtech/13176
Braun, V., & Clarke, V. (2006). Using thematic analysis in psychology. Qualitative Research in Psychology, 3(2), 77–101. https://doi.org/10.1191/1478088706qp063oa
Brown, D., & Cox, A. J. (2009). Innovative uses of video analysis. The Physics Teacher, 47(3), 145–150. https://pubs.aip.org/aapt/pte/article-abstract/47/3/145/275154/Innovative-Uses-of-Video-Analysis
Chi, M. T. H., Feltovich, P. J., & Glaser, R. (1981). Categorization and representation of physics problems by experts and novices. Cognitive Science, 5(2), 121–152. https://doi.org/10.1207/s15516709cog0502_2
Cotton, D. R. E., Cotton, P. A., & Shipway, J. R. (2024). Chatting and cheating: Ensuring academic integrity in the era of ChatGPT. Innovations in Education and Teaching International, 61(2), 228–239. https://doi.org/10.1080/14703297.2023.2190148
Crompton, H., & Burke, D. (2023). Artificial intelligence in higher education: The state of the field. International Journal of Educational Technology in Higher Education, 20, Article 22. https://doi.org/10.1186/s41239-023-00392-8
Dahlkemper, M. N., Lahme, S. Z., & Klein, P. (2023). How do physics students evaluate artificial intelligence responses on comprehension questions? A study on the perceived scientific accuracy and linguistic quality of ChatGPT. Physical Review Physics Education Research, 19(1), Article 010142. https://doi.org/10.1103/PhysRevPhysEducRes.19.010142
Docktor, J. L., & Mestre, J. P. (2014). Synthesis of discipline-based education research in physics. Physical Review Special Topics—Physics Education Research, 10(2), Article 020119. https://doi.org/10.1103/PhysRevSTPER.10.020119
Ennis, R. H. (1993). Critical thinking assessment. Theory Into Practice, 32(3), 179–186. https://doi.org/10.1080/00405849309543594
Facione, P. A. (1990). Critical thinking: A statement of expert consensus for purposes of educational assessment and instruction (ERIC Document No. ED315423). American Philosophical Association. https://eric.ed.gov/?id=ED315423
Gerlich, M. (2025). AI tools in society: Impacts on cognitive offloading and the future of critical thinking. Societies, 15(1), Article 6. https://doi.org/10.3390/soc15010006
Gregorcic, B., & Pendrill, A.-M. (2023). ChatGPT and the frustrated Socrates. Physics Education, 58(3), Article 035021. https://doi.org/10.1088/1361-6552/acc299
Große, C. S., & Renkl, A. (2007). Finding and fixing errors in worked examples: Can this foster learning outcomes? Learning and Instruction, 17(6), 612–634. https://doi.org/10.1016/j.learninstruc.2007.09.008
Halpern, D. F. (1998). Teaching critical thinking for transfer across domains: Dispositions, skills, structure training, and metacognitive monitoring. American Psychologist, 53(4), 449–455. https://doi.org/10.1037/0003-066X.53.4.449
Hong, Q. N., Fàbregues, S., Bartlett, G., Boardman, F., Cargo, M., Dagenais, P., Gagnon, M.-P., Griffiths, F., Nicolau, B., O’Cathain, A., Rousseau, M.-C., Vedel, I., & Pluye, P. (2018). The Mixed Methods Appraisal Tool (MMAT) version 2018 for information professionals and researchers. Education for Information, 34(4), 285–291. https://doi.org/10.3233/EFI-180221
Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y. J., Madotto, A., & Fung, P. (2023). Survey of hallucination in natural language generation. ACM Computing Surveys, 55(12), Article 248. https://doi.org/10.1145/3571730
Kasneci, E., Sessler, K., Küchemann, S., Bannert, M., Dementieva, D., Fischer, F., Gasser, U., Groh, G., Günnemann, S., Hüllermeier, E., Krusche, S., Kutyniok, G., Michaeli, T., Nerdel, C., Pfeffer, J., Poquet, O., Sailer, M., Schmidt, A., Seidel, T., … Kasneci, G. (2023). ChatGPT for good? On opportunities and challenges of large language models for education. Learning and Individual Differences, 103, Article 102274. https://doi.org/10.1016/j.lindif.2023.102274
Kohnke, L., Moorhouse, B. L., & Zou, D. (2023). ChatGPT for language teaching and learning. RELC Journal, 54(2), 537–550. https://doi.org/10.1177/00336882231162868
Kortemeyer, G. (2023). Could an artificial-intelligence agent pass an introductory physics course? Physical Review Physics Education Research, 19(1), Article 010132. https://doi.org/10.1103/PhysRevPhysEducRes.19.010132
Kregear, T., Babayeva, M., & Widenhorn, R. (2025). Analysis of student interactions with a large language model in an introductory physics lab setting. International Journal of Artificial Intelligence in Education, 35, 2993–3016. https://doi.org/10.1007/s40593-025-00489-3
Krupp, L., Steinert, S., Kiefer-Emmanouilidis, M., Avila, K. E., Lukowicz, P., Kuhn, J., Küchemann, S., & Karolus, J. (2024). Unreflected acceptance—Investigating the negative consequences of ChatGPT-assisted problem solving in physics education. In HHAI 2024: Hybrid human AI systems for the social good (Frontiers in Artificial Intelligence and Applications). IOS Press. https://doi.org/10.3233/FAIA240195
Küchemann, S., Steinert, S., Revenga, N., Schweinberger, M., Dinc, Y., Avila, K. E., & Kuhn, J. (2023). Can ChatGPT support prospective teachers in physics task development? Physical Review Physics Education Research, 19(2), Article 020128. https://doi.org/10.1103/PhysRevPhysEducRes.19.020128
Kuo, E., Hull, M. M., Gupta, A., & Elby, A. (2013). How students blend conceptual and formal mathematical reasoning in solving physics problems. Science Education, 97(1), 32–57. https://doi.org/10.1002/sce.21043
Lo, C. K. (2023). What is the impact of ChatGPT on education? A rapid review of the literature. Education Sciences, 13(4), Article 410. https://doi.org/10.3390/educsci13040410
Lubis, H., Van Harling, V. N., & Panunggul, V. B. (2025). Enhancing critical thinking in physics education through AI: A systematic literature review of trends and pedagogical implications. Jurnal Pendidikan dan Ilmu Fisika, 5(2), 343–354. https://doi.org/10.52434/jpif.v5i2.43353
Metcalfe, J. (2017). Learning from errors. Annual Review of Psychology, 68, 465–489. https://doi.org/10.1146/annurev-psych-010416-044022
Ng, D. T. K., Leung, J. K. L., Chu, S. K. W., & Qiao, M. S. (2021). Conceptualizing AI literacy: An exploratory review. Computers and Education: Artificial Intelligence, 2, Article 100041. https://doi.org/10.1016/j.caeai.2021.100041
OpenAI. (2023). GPT-4 technical report (arXiv:2303.08774). arXiv. https://doi.org/10.48550/arXiv.2303.08774
Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff, J. M., Akl, E. A., Brennan, S. E., Chou, R., Glanville, J., Grimshaw, J. M., Hróbjartsson, A., Lalu, M. M., Li, T., Loder, E. W., Mayo-Wilson, E., McDonald, S., … Moher, D. (2021). The PRISMA 2020 statement: An updated guideline for reporting systematic reviews. BMJ, 372, Article n71. https://doi.org/10.1136/bmj.n71
Parasuraman, R., & Manzey, D. H. (2010). Complacency and bias in human use of automation: An attentional integration. Human Factors, 52(3), 381–410. https://doi.org/10.1177/0018720810376055
Polverini, G., & Gregorcic, B. (2024a). How understanding large language models can inform the use of ChatGPT in physics education. European Journal of Physics, 45(2), Article 025701. https://doi.org/10.1088/1361-6404/ad1420
Polverini, G., & Gregorcic, B. (2024b). Performance of ChatGPT on the test of understanding graphs in kinematics. Physical Review Physics Education Research, 20(1), Article 010109. https://doi.org/10.1103/PhysRevPhysEducRes.20.010109
Polverini, G., Melin, J., Önerud, E., & Gregorcic, B. (2025). Performance of ChatGPT on tasks involving physics visual representations: The case of the brief electricity and magnetism assessment. Physical Review Physics Education Research, 21(1), Article 010154. https://doi.org/10.1103/PhysRevPhysEducRes.21.010154
Risko, E. F., & Gilbert, S. J. (2016). Cognitive offloading. Trends in Cognitive Sciences, 20(9), 676–688. https://doi.org/10.1016/j.tics.2016.07.002
Sallam, M. (2023). ChatGPT utility in healthcare education, research, and practice: Systematic review on the promising perspectives and valid concerns. Healthcare, 11(6), Article 887. https://doi.org/10.3390/healthcare11060887
Tlili, A., Shehata, B., Adarkwah, M. A., Bozkurt, A., Hickey, D. T., Huang, R., & Agyemang, B. (2023). What if the devil is my guardian angel: ChatGPT as a case study of using chatbots in education. Smart Learning Environments, 10, Article 15. https://doi.org/10.1186/s40561-023-00237-x
Tschisgale, P., Maus, H., Kieser, F., Kroehs, B., Petersen, S., & Wulff, P. (2025). Evaluating GPT- and reasoning-based large language models on Physics Olympiad problems: Surpassing human performance and implications for educational assessment. Physical Review Physics Education Research. https://doi.org/10.1103/6fmx-bsnl
UNESCO. (2024). AI competency framework for students. UNESCO. https://www.unesco.org/en/articles/ai-competency-framework-students
VanLehn, K. (2011). The relative effectiveness of human tutoring, intelligent tutoring systems, and other tutoring systems. Educational Psychologist, 46(4), 197–221. https://doi.org/10.1080/00461520.2011.611369
Wang, K. D., Burkholder, E., Wieman, C., Salehi, S., & Haber, N. (2024). Examining the potential and pitfalls of ChatGPT in science and engineering problem-solving. Frontiers in Education, 8, Article 1330486. https://doi.org/10.3389/feduc.2023.1330486
Zawacki-Richter, O., Marín, V. I., Bond, M., & Gouverneur, F. (2019). Systematic review of research on artificial intelligence applications in higher education – where are the educators? International Journal of Educational Technology in Higher Education, 16, Article 39. https://doi.org/10.1186/s41239-019-0171-0
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2026 Gladys Mahaut
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
CREAT: Contemporary Research in Education and Teaching publishes all articles under the Creative Commons Attribution–ShareAlike 4.0 International License (CC BY-SA 4.0) . This licensing policy supports the principles of open access, scholarly communication, knowledge dissemination, academic collaboration, and the responsible reuse of published research.
The application of the CC BY-SA 4.0 license allows readers, researchers, educators, institutions, and other users to access, reuse, share, and adapt the scholarly works published by CREAT: Contemporary Research in Education and Teaching, subject to the terms and conditions of the license. The policy is intended to maximize the accessibility and impact of scholarly knowledge while ensuring that the contributions of the original authors and the journal are properly recognized.
Under the Creative Commons Attribution–ShareAlike 4.0 International License (CC BY-SA 4.0), users are granted permission to copy, reproduce, distribute, display, perform, remix, transform, and build upon the published material in any medium or format. These permissions may be exercised for both non-commercial and commercial purposes, provided that the applicable requirements of the license are fulfilled.
The license is designed to facilitate the broad dissemination and reuse of scholarly knowledge while maintaining appropriate attribution to the original author(s) and publication. Users are responsible for ensuring that their use of the material complies with the terms of the CC BY-SA 4.0 license and applicable laws.
The CC BY-SA 4.0 license permits users to undertake the following activities:
Share: Users may copy and redistribute the published material in any medium or format. This includes sharing the work through educational platforms, institutional repositories, academic databases, websites, digital libraries, and other lawful channels.
Adapt: Users may remix, transform, translate, modify, and build upon the published material for any purpose, including commercial purposes, provided that appropriate attribution is given and the resulting adapted material is distributed in accordance with the ShareAlike requirement.
These permissions are granted for the duration of applicable copyright and related rights, subject to compliance with the conditions of the CC BY-SA 4.0 license.
Any reproduction, distribution, reuse, adaptation, translation, or other permitted use of material published by CREAT: Contemporary Research in Education and Teaching must provide appropriate credit to the original author(s).
Attribution should identify, where reasonably practicable, the name of the author(s), the title of the work, the journal in which the work was published, and the original publication or source. Users should also provide a link to the original work when such a link is reasonably available.
Users must provide a link to the CC BY-SA 4.0 license and clearly indicate whether any changes were made to the original material.
Attribution may be provided in any reasonable manner appropriate to the medium, context, and method of reuse. However, attribution must not be presented in a manner that suggests that the original author(s), CREAT: Contemporary Research in Education and Teaching, or its publisher endorses the subsequent use, adaptation, or user.
A fundamental condition of the CC BY-SA 4.0 license is the ShareAlike requirement. When a user creates an adaptation, transformation, translation, remix, or other derivative work based on material published by CREAT: Contemporary Research in Education and Teaching, the resulting material must be distributed under the same CC BY-SA 4.0 license or under a license identified by Creative Commons as compatible with the ShareAlike requirement.
The ShareAlike requirement is intended to ensure that adaptations and derivative scholarly works remain available for further access, reuse, and development under terms that preserve the same essential freedoms granted by the original license.
Users may not distribute an adapted version under terms that prevent subsequent users from exercising the permissions granted by the applicable ShareAlike license.
Users may not apply legal terms, contractual conditions, digital rights management systems, technological protection measures, or other restrictions that legally prevent others from exercising the freedoms granted by the CC BY-SA 4.0 license.
Any redistribution or adaptation must preserve the rights and freedoms granted under the applicable license. Users are expected to avoid practices that unnecessarily restrict access to the licensed material or its lawful adaptations.
Authors retain copyright and other applicable rights in their scholarly works, subject to the terms of the publication agreement established with CREAT: Contemporary Research in Education and Teaching. Publication under the CC BY-SA 4.0 license does not, by itself, transfer copyright ownership from the author(s) to the journal.
The CC BY-SA 4.0 license provides the public with permissions to reuse and adapt the published work while preserving appropriate recognition of the original author(s). Authors remain responsible for ensuring that their submitted work complies with applicable copyright, intellectual property, ethical, and legal requirements.
Not all materials appearing within a published article are necessarily covered by the CC BY-SA 4.0 license. Third-party content, including photographs, figures, tables, instruments, datasets, illustrations, quotations, or other copyrighted materials, may be subject to separate copyright or licensing conditions.
Where third-party material is included in an article and is not covered by the journal's CC BY-SA 4.0 license, the applicable rights and licensing conditions should be identified appropriately. Users intending to reuse such material are responsible for determining whether additional permission is required and for obtaining authorization from the relevant rights holder when necessary.
Users are not required to comply with the CC BY-SA 4.0 license for material that is already in the public domain or where a particular use is permitted by an applicable copyright exception or limitation.
Such uses remain subject to the applicable laws, regulations, and legal provisions governing the material. The existence of the CC BY-SA 4.0 license does not eliminate or restrict rights that users may independently possess under applicable copyright law.
The CC BY-SA 4.0 license does not necessarily grant permission to use personal data, publicity rights, privacy rights, trademarks, patents, or other rights that may be relevant to a particular use of the published material. Users are responsible for determining whether additional rights or permissions are required for their intended use.
The license also does not constitute a warranty that all permissions necessary for every possible use have been granted. Users should independently evaluate the legal, ethical, and academic implications of their intended reuse.
Although the CC BY-SA 4.0 license permits extensive reuse and adaptation, all users are expected to maintain academic integrity and responsible scholarly practice. Reuse, translation, adaptation, redistribution, or incorporation of published material must not misrepresent the original authorship, research methodology, findings, interpretation, or conclusions.
Any substantial modification, translation, adaptation, or transformation of published material should be clearly identified so that readers can distinguish the original work from the subsequent contribution.
Users are encouraged to preserve accurate bibliographic information and provide sufficient attribution to enable readers to locate and consult the original publication.
The adoption of the CC BY-SA 4.0 license reflects the commitment of CREAT: Contemporary Research in Education and Teaching to open scholarly communication and the broad dissemination of research findings. The license enables researchers, educators, students, institutions, and practitioners to access and reuse published knowledge while preserving attribution and the continued openness of adapted works.
Through this licensing policy, the journal encourages legitimate educational, academic, research, and professional uses of its published content and supports the development of new knowledge and scholarly contributions based on previously published research.
The complete terms and conditions of the Creative Commons Attribution–ShareAlike 4.0 International License (CC BY-SA 4.0) are available through the official Creative Commons website:
Creative Commons Attribution–ShareAlike 4.0 International (CC BY-SA 4.0)
Users are encouraged to review the complete license terms before undertaking substantial reproduction, redistribution, adaptation, translation, commercial use, or other reuse of articles published by CREAT: Contemporary Research in Education and Teaching.
By publishing under the Creative Commons Attribution–ShareAlike 4.0 International License (CC BY-SA 4.0), CREAT: Contemporary Research in Education and Teaching supports open access, responsible scholarly communication, broad dissemination of research, educational reuse, academic collaboration, and the development of derivative scholarly works, while requiring appropriate attribution and preservation of the same licensing freedoms for adaptations.
The journal expects all authors, readers, researchers, educators, institutions, and other users to respect the terms of the CC BY-SA 4.0 license and to uphold the principles of academic integrity, transparency, responsible attribution, and ethical scholarly communication.
