A Multi-Account Statistical Evaluation of ChatGPT Proficiency in the Kurdish Sorani Language
DOI:
https://doi.org/10.21928/uhdjst.v9n2y2025.pp319-334Keywords:
ChatGPT, Chatbot, AI, Kurdish Language, Sorani Dialect, Statistical AnalysisAbstract
This research analyzes the strengths and weaknesses of ChatGPT in responding to questions posed in the Kurdish language, specifically its Sorani dialect, by evaluating its responses to a structured dataset of 50 multiple-choice questions across multiple topics such as language, history, culture, and general knowledge. Using four independent user accounts, each subjected to ten repeated testing cycles, the research assesses accuracy, consistency, and variation influenced by account identity, session timing, and model behavior. This study evaluates the multilingual capabilities of ChatGPT by comparing its performance in Kurdish (Sorani) and Arabic languages. The research establishes a framework to examine how artificial intelligence chatbots, such as ChatGPT, function as applied tools for language understanding and educational use. The analysis demonstrates that ChatGPT achieved an overall average accuracy rate of approximately 70%, indicating satisfactory performance in multilingual contexts. However, significant variations were observed across different user accounts, suggesting that factors such as user profile and temporal dynamics can considerably influence output consistency. The comparative findings highlight the developmental challenges in Arabic and Kurdish language processing, emphasizing the need for further refinement of ChatGPT’s linguistic performance and its effective integration into academic and technological applications. While ChatGPT exhibited proficiency in answering general knowledge questions, it demonstrated a limited understanding of specialized topics in Kurdish, particularly classical literature and historical content. The research presents the strengths and limitations of ChatGPT for under-resourced languages and provides feedback to developers, educators, and researchers. Observing patterns in accuracy, question difficulty, and error behavior, this research also contributes to ongoing efforts toward improving the linguistic and cultural adequacy of AI models for under-resourced languages.
References
N. Rane. “ChatGPT and similar generative artificial intelligence (AI) for smart industry: Role, challenges and opportunities for industry 4.0, industry 5.0 and society 5.0”. SSRN Electronic Journal, vol. 2, no. 1, pp. 10–17, 2024.
K. Ofosu-Ampong. “Artificial intelligence research: A review on dominant themes, methods, frameworks and future research directions”. Telematics and Informatics Reports, vol. 14, p. 100127, 2024.
M. J. Sousa, S. Pani, F. dal Mas and S. Sousa. “Incorporating AI Technology in the Service Sector. Apple Academic Press, New York. 2024.
M. M. Maas. “Concepts in advanced AI governance: A literature review of key terms and definitions”. AI Foundations Report, vol. 3, 2023.
A. Casheekar, A. Lahiri, K. Rath, K. S. Prabhakar and K. Srinivasan. “A contemporary review on chatbots, AI-powered virtual conversational agents, ChatGPT: Applications, open challenges and future research directions”. Computer Science Review, vol. 52, p. 100632, 2024.
A. Mary Sowjanya and K. Srividya. ividyaSrividyaSrividya24. emConversational Artificial Intelligence. Wiley, New Jersey. pp. 713-725, 2024.
D. Das, C. Prasad and J. Geetha. “Intelligent Conversational AI for Microsoft Teams with Actionable Insights”. In: 2024 8th International Conference on Computational System and Information Technology for Sustainable Solutions (CSITSS). IEEE. pp. 1-5, 2024.
L. Jain, R. Ananthasayam, U. Gupta and R. Radha. “Comparison of rule-based chat bots with different machine learning models”. Procedia Computer Science, vol. 259, pp. 788-798, 2025.
E. Yurdakurban, K. G. Topsakal and G. S. Duran. “A comparative analysis of AI-based chatbots: Assessing data quality in orthognathic surgery related patient information”. Journal of Stomatology Oral and Maxillofacial Surgery, vol. 125, no. 5, p. 101757, 2024.
A. M. Saghiri, S. M. Vahidipour, M. R. Jabbarpour, M. Sookhak and A. Forestiero. “A survey of artificial intelligence challenges: Analyzing the definitions, relationships, and evolutions”. Applied Sciences, vol. 12, no. 8, p. 4054, 2022.
Y. Dong, J. Hou, N. Zhang and M. Zhang. “Research on how human intelligence, consciousness, and cognitive computing affect the development of artificial intelligence”. Complexity, vol. 2020, pp. 1-10, 2020.
I. H. Sarker. “AI-based modeling: Techniques, applications and research issues towards automation, intelligent and smart systems”. SN Computer Science, vol. 3, no. 2, p. 158, 2022.
A. P. Chaves, J. Egbert, T. Hocking, E. Doerry and M. A. Gerosa. “Chatbots language design: The influence of language variation on user experience with tourist assistant chatbots”. ACM Transactions on Computer-Human Interaction, vol. 29, no. 2, pp. 1-38, 2022.
K. Mageira, D. Pittou, A. Papasalouros, K. Kotis, P. Zangogianni and A. Daradoumis. “Educational AI Chatbots for content and language integrated learning”. Applied Sciences, vol. 12, no. 7, p. 3239, 2022.
A. Bewersdorff, K. Seßler, A. Baur, E. Kasneci and C. Nerdel. “Assessing student errors in experimentation using artificial intelligence and large language models: A comparative study with human raters”. Computers and Education: Artificial Intelligence, vol. 5, p. 100177, 2023.
N. Haristiani. “Artificial intelligence (AI) chatbot as language learning medium: An inquiry”. The Journal of Physics: Conference Series, vol. 1387, no. 1, p. 012020, 2019.
A. S. E. AbuSahyon, A. Alzyoud, O. Alshorman and B. Al-Absi. “AI-driven technology and chatbots as tools for enhancing english language learning in the context of second language acquisition: A review study”. International Journal of Membrane Science and Technology, vol. 10, no. 1, pp. 1209-1223, 2023.
I. Beltagy, A. Cohan, R. Logan IV, S. Min and S. Singh. “Zero- and Few-Shot NLP with Pretrained Language Models”. In: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics: Tutorial Abstracts, Stroudsburg, PA, USA: Association for Computational Linguistics. pp. 32-37, 2022.
T. Shin, Y. Razeghi, R. L. Logan IV, E. Wallace and S. Singh. “Autoprompt: Eliciting knowledge from language models with automatically generated prompts”. arXiv preprint arXiv:2010.15980, 2020.
X. V. Lin, T. Mihaylov, M. Artetxe, T. Wang, S. Chen, D. Simig, M. Ott, N. Goyal, S. Bhosale, J. Du, R. Pasunuru,… & X. Li. “Few-shot learning with multilingual language models,” arXiv preprint arXiv:2112.10668, 2021. doi: 10.48550/arXiv.2112.10668
H. Naveed, A. U. Khan, S. Qiu, M. Saqib, S. Anwar, M. Usman, N. Akhtar, N. Barnes and A. Mian. “A comprehensive overview of large language models”. ACM Transactions on Intelligent Systems and Technology, vol. 16, no. 5, pp. 1-72, 2025.
B. K. Arif and A. M. Aladdin. “A comparative analysis of ChatGPT and traditional machine learning algorithms on real-world data”. Kurdistan Journal of Applied Research, vol. 10, no. 2, pp. 93-118, 2025.
E. A. Alomari. “Unlocking the potential: A comprehensive systematic review of ChatGPT in natural language processing tasks”. Computer Modeling in Engineering and Sciences, vol. 141, no. 1, pp. 43-85, 2024.
A. Asai, S. Kudugunta, X. Yu, T. Blevins, H. Gonen, M. Reid, Y. Tsvetkov, S. Ruder and H. Hajishirzi. “BUFFET: Benchmarking Large Language Models for Few-shot Cross-lingual Transfer”. In: Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Vol. 1: Long Papers. Association for Computational Linguistics, Stroudsburg, PA, USA, pp. 1771-1800, 2024.
P. Efimov, L. Boytsov, E. Arslanova and P. Braslavski. “The Impact of Cross-Lingual Adjustment of Contextual Word Representations on Zero-Shot Transfer”. Springer, Berlin. pp. 51-67, 2023.
E. Razumovskaia, I. Vulić and A. Korhonen. “Data augmentation and learned layer aggregation for improved multilingual language understanding in dialogue”. In: Findings of the Association for Computational Linguistics: ACL 2022, Stroudsburg, PA, USA. pp. 2017-2033, 2022.
W. Huang, K. F. Hew and L. K. Fryer., 2022.l Linguistics: ACL 2022tilingual language understanding in dialogue”. In: sks”. ta”. uage acqguage acqe acqagJournal of Computer Assisted Learning, vol. 38, no. 1, pp. 237-257, 2022.
G. A. Santos, G. G. de Andrade, G. R. S. Silva, F. C. M. Duarte, J. P. J. Da Costa and R. T. de Sousa. “A conversation-driven approach for chatbot management”. IEEE Access, vol. 10, pp. 8474-8486, 2022.
W. Maeng and J. Lee. “Designing a Chatbot for survivors of sexual violence: Exploratory study for hybrid approach combining rule-based chatbot and ML-based Chatbot”. In: Asian CHI Symposium 2021. ACM, New York, NY, USA. pp. 160-166, 2021.
A. Sakshi, T. Mehrotra, P. Tyagi, and V. Jain, “Emerging trends in hybrid information systems modeling in artificial intelligence”. In: Hybrid Information Systems. De Gruyter, Germany. pp. 115- 152, 2024.
M. Orosoo, I. Goswami, F. R. Alphonse, G. Fatma, M. Rengarajan and B. Kiran Bala. “Enhancing Natural Language Processing in Multilingual Chatbots for Cross-Cultural Communication”. In: 2024 5th International Conference on Intelligent Communication Technologies and Virtual Mobile Networks (ICICV). IEEE. pp. 127-133, 2024.
J. Guerrero-Ibáñez, S. Zeadally and J. Contreras-Castillo. “Sensor technologies for intelligent transportation systems”. Sensors, vol. 18, no. 4, p. 1212, 2018.
H. Leunard, R. Rachmawati, B. N. Zani and K. Maharjan. “GPT Chat: Opportunities and challenges in the learning process of Arabic language in higher education”. Journal International of Lingua and Technology, vol. 2, no. 1, p. 10, 2023.
A. M. Aladdin, Y. N. Bakir and S. I. Saeed. “The effects to trend the suitable OS platform”. Journal of Advances in Natural Sciences, vol. 5, no. 1, pp. 342-351, 2018.
V. İnci Kavak, D. Evis and A. Ekinci. “The use of ChatGPT in language education”. Experimental and Applied Medical Science, vol. 5, no. 2, pp. 72-82, 2024.
F. Mosaiyebzadeh, S. Pouriyeh, R. Parizi, N. Dehbozorgi, M. Dorodchi and D. Macêdo Batista. “Exploring the Role of ChatGPT in Education: Applications and Challenges”. In: The 24th Annual Conference on Information Technology Education. ACM, New York, NY, USA, pp. 84-89, 2023.
V. Goar, N. S. Yadav and P. S. Yadav. “Conversational AI for natural language processing: An review of ChatGPT”. International Journal on Recent and Innovation Trends in Computing and Communication, vol. 11, no. 3s, pp. 109-117, 2023.
G. Sharma and A. Thakur. “ChatGPT in drug discovery”. ChemRxiv, 2023.
F. Fui-Hoon Nah, R. Zheng, J. Cai, K. Siau and L. Chen. “Generative AI and ChatGPT: Applications, challenges, and AI-human collaboration”. Journal of Information Technology Case and Application Research, vol. 25, no. 3, pp. 277-304, 2023.
A. M. Aladdin, R. K. Muhammed, H. S. Abdulla and T. A. Rashid. “ChatGPT: Precision Answer Comparison and Evaluation Model”. 2024. DOI: 10.36227/techrxiv.172833414.47483047/v1
L. Kohnke, B. L. Moorhouse and D. Zou. “ChatGPT for language teaching and learning”. RELC Journal, vol. 54, no. 2, pp. 537-550, 2023.
J. S. Barrot. “Using ChatGPT for second language writing: Pitfalls and potentials”. Assessing Writing, vol. 57, p. 100745, 2023.
P. Zangrossi, M. Martini, F. Guerrini, P. De Bonis and G. Spena. “Large language model, AI and scientific research: Why ChatGPT is only the beginning”. The Journal of Neurosurgical Sciences, vol. 68, no. 2, 2024.
B. F. Gonçalves and V. Gonçalves. “Artificial Intelligence Language Models: The Path to Development or Regression for Education?” Springer, Berlin. pp. 56-65, 2024.
D. S. M. Pereira, F. Mourão, J. C. Ribeiro, P. Costa, S. Guimarães and J. M. Pêgo. “ChatGPT as an item calibration tool: Psychometric insights in a high-stakes examination”. MedTeach, vol. 47, no. 4, pp. 677-683, 2025.
M. K. Audichya and J. R. Saini. “ChatGPT for Creative Writing and Natural Language Generation in Poetry and Prose”. In: 2023 International Conference on Advanced Computing Technologies and Applications (ICACTA). IEEE. pp. 1-7, 2023.
A. Azaria. “ChatGPT: More Human-Like Than Computer-Like, but Not Necessarily in a Good Way”. In: 2023 IEEE 35th International Conference on Tools with Artificial Intelligence (ICTAI). IEEE. pp. 468-473, 2023.
H. Al-Khatri, N. Al-Azri, H. Hassan, R. A. Maamari and D. A. Khatri, “ChatGPT applications in active learning in higher education: A restricted systematic review,” In: Proceedings of the 11th International Conference Higher Education Advances (HEAd), 2025. Available from: https://ocs.editorial.upv.es/index.php/HEAD/ HEAd25/paper/view/20029
D. O. Hasan, A. M. Aladdin, A. A. H. Amin, T. A. Rashid, Y. H. Ali, M. Al-Bahri, J. Majidpour, I. Batrancea and E. S. Masca. “Perspectives on the impact of E-Learning pre- and post-COVID-19 pandemic-the case of the Kurdistan Region of Iraq”. Sustainability, vol. 15, no. 5, p. 4400, 2023.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2025 Alla Ahmad Hassan, Hemin Sardar Abdulla, Tara Yousif Mawlood, Rebwar Khalid Muhammed, Aso M. Aladdin, Tarik A. Rashid
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
