Assessing the Consistency of AI, Peer, Self, and Expert Assessments in Evaluating Graduation Projects
DOI:
https://doi.org/10.52131/jer.2025.v6i2.2925Keywords:
AI in Assessment, Self-Assessment, Peer Assessment, Authentic Assessment, Experts’ Assessment, Educational TechnologyAbstract
The rapid spread of artificial intelligence and modern trends towards using its applications in assessment processes that have become directed towards authentic assessment have prompted researchers to pay attention to investigating the consistency of the results of the authentic assessment of experts as a criterion with the results of several other assessments (artificial intelligence, self, and peer assessment) and determining the assessment method that is most consistent with expert assessments. This research also aimed to study the differences between the results of the authentic assessment of students' graduation projects using different assessment methods. To examine this, the researchers used a rubric that was prepared in a way that suits the graduation project standards set by a group of specialists, and based on the descriptive quantitative approach with a cross-sectional design, where a group of graduation projects of students from the Faculty of Education at An-Najah National University in Palestine were evaluated as a random cluster sample by experts as a criterion and assessment using Chat-GPT, peer assessment, and self-assessment. The results showed that there was a strong consistency between the expert assessment and Chat-GPT assessment, with a Pearson correlation coefficient of R=0.897. Compared to the peer assessment, the results show no meaningful alignment with the expert assessment, where the correlation coefficient is R=0.380. Unlike the peer assessment, the results show weak alignment to the expert evaluation with R = 0.380. The selfassessment, in contrast, shows a moderate alignment with the expert assessment, with a Pearson correlation of R = 0.484. From the above, it can be inferred that, among the other methods, the AI-based assessment had the highest alignment with expert judgment.
References
References
Al Maktoum, S. B., & Al Kaabi, A. M. (2024). Exploring teachers’ experiences within the teacher assessment process: A qualitative multi-case study. Cogent Education, 11(1), 2287931.
Awidi, I. T. (2024). Comparing expert tutor assessment of reflective essays with marking by generative artificial intelligence (AI) tool. Computers and Education: Artificial Intelligence, 6, 100226.
Ayyoub, A. A., Assali, A., Eideh, B. A., & Suleiman, M. (2017). An Action Research Approach for Using Self/Peer Assessment to Enhance Learning and Teaching Outcomes. Journal of Teaching and Teacher Education, 5(01), 33–42.
Ayyoub, A. A., Bsharat, A., & Suleiman, M. (2021). The impact of alternative authentic assessment outcomes in Palestinian fourth grade math classrooms. Studies in Educational Assessment, 70(May), 101056. https://doi.org/10.1016/j.stueduc.2021.101056
Bower, M., Torrington, J., Lai, J. W. M., Petocz, P., & Alfano, M. (2024). How should we change teaching and assessment in response to increasingly powerful generative Artificial Intelligence? Outcomes of the ChatGPT teacher survey. Education and Information Technologies, 1–37.
Butler, Y. G. (2024). Self-assessment in second language learning. Language Teaching, 57(1), 42–56.
Castillo-Martínez, I. M., Ramírez-Montoya, M. S., Glasserman-Morales, L. D., & Millán-Arellano, J. A. (2024). eComplex: validity and reliability of rubric for assessing reasoning for complexity competency. Quality & Quantity, 58(2), 1545–1563.
Chunping, Z., Xu, C., Huayang, Z., & Ching Sing, C. (2024). Automated versus Peer Assessment: Effects on Learners’ English Public Speaking. Language Learning & Technology, 28(2), 210–228. https://search.ebscohost.com/login.aspx?direct=true&db=eric&AN=EJ1427980&site=ehost-live
Coskun, T. K., & Alper, A. (2024). Evaluating the evaluators: A comparative study of AI and teacher assessments in Higher Education. Digital Education Review, 45, 124–140. https://doi.org/10.1344/der.2024.45.124-140
Dasgupta, A. P., Anderson, T. R., & Pelaez, N. (2014). Development and validation of a rubric for diagnosing students’ experimental design knowledge and difficulties. CBE Life Sciences Education, 13(2), 265–284. https://doi.org/10.1187/cbe.13-09-0192
De La Rosa Gómez, A., Cano, J. M. M., & Díaz, G. A. M. (2019). Validation of a rubric to evaluate open educational resources for learning. Behavioral Sciences, 9(12). https://doi.org/10.3390/bs9120126
Dhara, S., Chatterjee, S., Chaudhuri, R., Goswami, A., & Ghosh, S. K. (2022). Artificial Intelligence in Assessment of Students’ Performance. In Artificial Intelligence in Higher Education (pp. 153–167). CRC Press.
Dimitriadou, E., & Lanitis, A. (2023). A critical assessment, challenges, and future perspectives of using artificial intelligence and emerging technologies in smart classrooms. Smart Learning Environments, 10(1), 12.
Hodges, C. B., & Kirschner, P. A. (2024). Innovation of instructional design and assessment in the age of generative artificial intelligence. TechTrends, 68(1), 195–199.
Hooda, M., Rana, C., Dahiya, O., Rizwan, A., & Hossain, M. S. (2022). Artificial intelligence for assessment and feedback to enhance student success in higher education. Mathematical Problems in Engineering, 2022(1), 5215722.
Hu, J. (2021). Teaching Assessment System by use of Machine Learning and Artificial Intelligence Methods. International Journal of Emerging Technologies in Learning (IJET), 16(5), 87–101. https://www.learntechlib.org/p/220079
Iglesias Pérez, M. C., Vidal-Puga, J., & Pino Juste, M. R. (2022). The role of self and peer assessment in Higher Education. Studies in Higher Education, 47(3), 683–692.
Ismail, S. M., & Heydarnejad, T. (2023). Probing into the influence of EFL learners’ self-assessment and assessment apprehension in predicting their personal best goals and self-efficacy skills: a structural equation modeling. Language Testing in Asia, 13(1), 8.
Kadri, N., & Amziane, H. (2021). Students’ Attitudes about Self-Assessment: A Neglected Aspect in the Algerian EFL Classrooms. The Educational Review, USA, 5(8), 275–286.
London, M., Sessa, V. I., & Shelley, L. A. (2023). Developing self-awareness: Learning processes for self-and interpersonal growth. Annual Review of Organizational Psychology and Organizational Behavior, 10(1), 261–288.
Mao, J., Chen, B., & Liu, J. C. (2024). Generative Artificial Intelligence in Education and Its Implications for Assessment. TechTrends, 68(1), 58–66. https://doi.org/10.1007/s11528-023-00911-4
Milic, S., & Simeunovic, V. (2022). Concordance between Giftedness Assessments by Teachers, Parents, Peers and the Self-Assessment Using Multiple Intelligences. In High Ability Studies (Vol. 33, Issue 1, pp. 1–19).
Norova, M. (2020). Advantages And Disadvantages of Traditional and Alternative Ways of Assessment. ????? ??????? ?????????? (Buxdu. Uz), 2(2).
Palm, T. (2019). Performance assessment and authentic assessment: A conceptual analysis of the literature. Practical Assessment, Research, and Assessment, 13(1), 4.
Planas-Lladó, A., Feliu, L., Arbat, G., Pujol, J., Suñol, J. J., Castro, F., & Mart’i, C. (2021). An analysis of teamwork based on self and peer assessment in higher education. Assessment & Assessment in Higher Education, 46(2), 191–207.
Power, J. R., & Tanner, D. (2023). Peer Assessment, Self-Assessment, and Resultant Feedback: An Examination of Feasibility and Reliability. In European Journal of Engineering Education (Vol. 48, Issue 4, pp. 615–628).
Rudolph, J., Tan, S., & Tan, S. (2023). War of the chatbots: Bard, Bing Chat, ChatGPT, Ernie and beyond. The new AI gold rush and its impact on higher education. Journal of Applied Learning and Teaching, 6(1), 364–389.
Shabara, R., ElEbyary, K., & Boraie, D. (2024). TEACHERS or CHATGPT: The ISSUE of ACCURACY and CONSISTENCY in L2 ASSESSMENT. Teaching English with Technology, 24(2), 71–92. https://doi.org/10.56297/vaca6841/LRDX3699/XSEZ5215
Stevens, D. D., & Levi, A. J. (2023). Introduction to rubrics: An assessment tool to save grading time, convey effective feedback, and promote student learning. Routledge.
Swiecki, Z., Khosravi, H., Chen, G., Martinez-Maldonado, R., Lodge, J. M., Milligan, S., Selwyn, N., & Gaševi?, D. (2022). Assessment in the age of artificial intelligence. Computers and Education: Artificial Intelligence, 3, 100075.
Taylor, B., Kisby, F., & Reedy, A. (2024). Rubrics in higher education: an exploration of undergraduate students’ understanding and perspectives. Assessment & Assessment in Higher Education, 49(6), 799–809. URL: https://doi.org/10.1080/02602938.2023.2299330
Tomczyk, ?., Limone, P., & Guarini, P. (2024). Assessment of modern educational software and basic digital competences among teachers in Italy. Innovations in Education and Teaching International, 61(2), 355–369.
Vogt, W. P., Gardner, D. C., & Haeffele, L. M. (2012). When to use what research design. Guilford Press.
Wahyudin, A. Y., Pustika, R., & Simamora, M. W. B. (2021). Vocabulary learning strategies of EFL students at tertiary level. The Journal of English Literacy Education: The Teaching and Learning of English as a Foreign Language, 8(2), 101–112.
Yan, Z., Brown, G. T. L., Lee, J. C.-K., & Qiu, X.-L. (2020). Student self-assessment: Why do they do it? Educational Psychology, 40(4), 509–532.
Yan, Z., Lao, H., Panadero, E., Fernández-Castilla, B., Yang, L., & Yang, M. (2022). Effects of self-assessment and peer-assessment interventions on academic performance: A meta-analysis. Educational Research Review, 37, 100484.
Yan, Z., Panadero, E., Wang, X., & Zhan, Y. (2023). A systematic review on students’ perceptions of self-assessment: usefulness and factors influencing implementation. Educational Psychology Review, 35(3), 81.
Yan, Z., Wang, X., Boud, D., & Lao, H. (2023). The effect of self-assessment on academic performance and the role of explicitness: a meta-analysis. Assessment & Assessment in Higher Education, 48(1), 1–15. URL: https://doi.org/10.1080/02602938.2021.2012644
Yu, H. (2024). The application and challenges of ChatGPT in educational transformation: New demands for teachers’ roles. Heliyon.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2025 Haneen Taqatqa, Doaa Hakawati, Suad Qurini, Hussam Daood, Abdalkarim Ayyoub
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
