A Multi-Account Statistical Evaluation of ChatGPT Proficiency in the Kurdish Sorani Language

2.1. Performance Evaluation of AI Tools

Although few-shot English learning has been advanced by in-context learning methods that do not update parameters [18], [19], few-shot cross-lingual transfer remains uncharted [20]. While encouraging results in non-English languages are seen [21], [22], [23]. However, it remains unclear whether these in-context learning methods can outperform traditional fine-tuning approaches across different languages and task types. This reflects a chronic lack of balance in NLP research, with English-biased development dominating and limiting development in low-resource and multilingual settings.

The BUFFET benchmark (Benchmark of Unified Format Few-shot Transfer Evaluation) attempts to mitigate this by evaluating few-shot transfer on 15 tasks across 54 languages [24]. Nevertheless, existing challenges such as data paucity, linguistic diversity, and test bias [25] [26] limit unbiased evaluation. Thus, advancement on multilingual benchmarks like ChatGPT still remains crucial to achieve unbiased performance beyond English.

In NLP, chatbots focus on parsing user input and generating responses, particularly by controlling prior knowledge. Several bots analyze the keywords and phrases used by users and gradually learn to respond more accurately and relevantly over time. In a chatbot system, it is not necessary for users to speak using the same dialect or linguistic structure as the chatbot itself [27]. When a user submits a message, the chatbot must interpret and handle the input [28]. However, since the user’s language is often natural and unstructured, the chatbot requires a means to translate this input into a structured format that a machine can comprehend and act upon. As a result, this procedure is known as natural language understanding, as detailed under the dialect language component in Table 1. The conversion of a user’s communication into structured data is referred to as constructing a semantic frame. This entire operation requires numerous steps, which are depicted in a general workflow in Fig. 1.

TABLE 1: Essential elements in chatbot design for multilingual and dialect-specific communication

Fig. 1. The key elements involved in chatbot conversation processing.

Furthermore, hybrid Chatbot [29] is a capable conversational agent which combines the advantages of rule-based systems and AI-driven approaches to develop a stronger and more versatile user experience. It combines accuracy and consistency achieved through the application of predefined rules for dealing with structured queries, with the understanding and responding capabilities to complex unstructured queries made possible by ML and NLP [30]. The combination of the two provides the chatbot with the ability to manage both types of interactions, from static FAQs to a more engaging, responsive, and contextually aware dialog, all while being more efficient, scalable, and providing better user satisfaction.

To develop a multilingual chatbot with dialectal support, the system must include automatic language-detection modules, integrate translation APIs (Application Programming Interface), and be trained on datasets that contain diverse linguistic and dialect-specific samples [31]. Regional idioms and dialectal variations should be integrated through NLP methods that enable appropriate comprehension and response within the parameters of the idiom or dialect from which they are expressed. To create or at least simulate a chatbot that understands dialects, it must possess the important elements given in Table 1.

As a result, the advantages and disadvantages of chatbot tools across different types have been recently surveyed. These studies highlight their potential to enhance personalized learning and student engagement. However, they also raise concerns, particularly regarding accuracy, cultural relevance, and ethical considerations.

2.2. ChatGPT Applications and Critical Analysis in Literature

The use of ChatGPT, an AI language model created by OpenAI, in Kurdish language education is a double-edged sword. Although ChatGPT performs well in languages with larger online corpora, its performance in less-represented languages, such as Kurdish, has not been as effective. It can serve as an autodidactic device, a dialog partner, and a creator of stimulating learning materials. This said, one should also be mindful of the challenges posed by engaging Chat GPT around human interaction, culture, mistakes and fixes, personal feedback [32]. Leveraging ChatGPT as a tool for learning the Arabic language in post-secondary education would provide fruitful opportunities in terms of learner experience on several platforms and within the existing operating systems and in terms of efficiency overall [33], [34].

It shows how, despite having the potential to improve language teaching by providing personalized feedback and tailored lessons, challenges posed by misinformation, absence of human interaction, and cultural sensitivity need to be solved, especially for less represented languages such as Kurdish [35]. Plus, in addition to the advantages of personalized learning and reduced tasks in education, ChatGPT is also a challenge due to the possibilities of academic dishonesty and issues related to identifying AI-generated text, thus threatening students’ abilities to solve problems [36].

ChatGPT is an application of the transformer model for dialog generation and other NLP applications, such as question answering, developed by OpenAI. This is a language model for NLP tasks such as language generation developed by OpenAI [37]. It is capable of producing human-like conversational text. It is automatically generated, without human conversation or participation, chat in natural language, but it does not have human levels of understanding, empathy, or creativity [38]. Hence, ChatGPT is an OpenAI’s state-of-the-art natural language model for natural conversational applications [39]. These applications and models, such as customer service, virtual assistants, and comparisons, use deep learning and NLP to provide human-like responses [39], [40]. ChatGPT, a generative AI chatbot, is another tool and resource that can aid language teaching and learning but must be used ethically and effectively with ethical and effective digital skills [41].

While ChatGPT may serve as a promising writing tutor for second language writing, more research is needed on the implications for writing pedagogy and for academic integrity [42]. Conversely, ChatGPT is a humanlike conversational agent based on the GPT architecture. Despite its limitations and ethical issues, it assists scientific studies by providing reviews of the literature, summaries of the content, and decisions in areas such as medicine and surgery [43].

The study examines the artificial language model ChatGPT, which generates human-like discursive text using deep-learning processes, and was developed by OpenAI. It engages as well with the opportunities and dangers associated with its use in an educational context [44]. The other looks into the use of ChatGPT in the item editing and modification process of high-stake testing, highlighting its use as an automated solution to a process within education and testing that essentially leads to enormous efficiency and cost benefits [45]. The model under investigation is ChatGPT, an OpenAI language model used for creative writing in poetry and prose. It explores how it works, what it takes into account when generating text, and the challenges of using it as a tool for creativity [46]. ChatGPT is a large language model for conversation developed by OpenAI. It is also human-like and biased because it has been trained on data generated by humans and then fine-tuned based on human feedback, which leads it to generate responses that are very much in line with what humans prefer and human cognitive biases [47]. The study analyses the application of GPT language models, namely ChatGPT, in higher education, focusing on their application as supports for engaging in theoretical exercises and laboratory practices and the questions of use and assessment in education. Effectiveness declines from theory to exercises to labs, as GPT struggles with specialized tools and software [48].

Critically, these literature studies present a balanced yet fragmented understanding of ChatGPT’s role in education. Collectively, the studies highlight both its pedagogical benefits and ethical challenges. They emphasize potential for personalized learning, efficiency, and engagement but also reveal persistent limitations in accuracy, cultural sensitivity, and academic integrity. Overall, the evaluations remain exploratory, requiring deeper empirical validation across diverse linguistic contexts.

3.1. Data Collection

The study’s dataset consists of 50 unique multiple-choice questions taken from a number of Kurdish language books. These questions are about geography, history, language structure, culture, and politics. There are four possible answers to each question, one of which is the actual answer. The data were handpicked to be culturally and linguistically relevant with the purpose of being useful for language assessment and educational use. The questions were hand-picked and screened for clarity, cultural, and linguistic appropriateness by native speakers of Kurdish in the Sorani dialect.

3.2. Data Preparation

Questions were assessed for clarity as well as for consistency in formatting. Response options were identified using the Kurdish alphabet (a,b,c,d), and all text was rendered in Unicode to help the display of the Kurdish script properly. The questions were entered into a database in a structured format suitable for processing and analysis.

The present study employs already established assessment frameworks through four distinct test accounts to measure ChatGPT’s performance in the Sorani dialect Kurdish language. Each account contains 50 diverse questions that test different aspects of the knowledge of the Kurdish language. To obtain reliable, robust data, each question was asked ten times. The procedures for question formulation and the criteria employed for evaluating the responses are comprehensively outlined in the following section.

Fig. 2 shows the workflow of the experimental design to evaluate ChatGPT’s performance in answering Kurdish (Sorani) questions. It starts with four independent ChatGPT accounts (A, B, C, D), which all receive the same set of 50 questions. Each account goes through the processing of the questions over several testing sessions. All the responses are kept in separate data sets and statistical analyses are performed on it. The result is an analysis of ChatGPT’s correctness, types of errors, and consistency of performance, which offers new observations regarding how knowledge is represented by the model in the context of a low-resource language.

Fig. 2. Multi-account testing and data collection workflow.

3.3. Experimental Design

To analyze the consistency and reliability of ChatGPT, four separate ChatGPT accounts (A, B, C, and D) were used in the study. Both accounts underwent ten test runs on the complete dataset of 50 questions. This resulted in a total of 2000 responses (4 accounts × 10 sessions [Rounds] × 50 questions). Testing for each iteration occurred at varying times of day for each account over a total period of five days. Tests were either scheduled for morning hours (between 6 AM and 9 AM GMT) or evening hours (between 6 PM and 12 AM GMT). Questions were presented in a randomized order for each session to prevent any learning or memorization effects. The answers were all transcribed in a file for each account and session, which were ChatGPT’s responses in each dialect of the Kurdish language. The questions, as noted above, were scientific, literary, and geographical. The process is depicted in Fig. 3. Then, the next step is to conduct the evaluation, which is described in the next sections, in terms of how the responses perform and what they look like.

Fig. 3. The multiuser framework for question answering in Sorani dialect with ChatGPT.

3.3.1. Question-answer evaluation based on test rounds collections

As determined, the process begins with the initialization of four independent accounts (A, B, C, and D), each configured to perform the same set of tasks under identical conditions. The core of the research involves a loop that repeats ten times, ensuring consistency and reliability throughout the repeated testing process. Within each loop iteration, all four accounts simultaneously process a standardized set of 50 questions, designed to evaluate AI performance. The method reflects a systematic, repeatable testing structure that enables both cross-account comparison and longitudinal performance evaluation, contributing to the methodological rigor of the study.

According to this, to evaluate the responses generated by ChatGPT, a set of fifty questions was used, distributed across 40 test sessions conducted through four different user accounts. These sessions are used to assess the average performance of ChatGPT, with attention given to identifying any variations based on the time of testing or the specific user account. The evaluation results are presented using equation 1 as the average score per session or rounds, which highlights the performance outcomes for each individual test session.

In this evaluation, n represents the total number of test sessions conducted, while T_i refers to the number of correct answers provided by ChatGPT during the i–th session. Each session corresponds to one instance of testing, and the value of T_i reproduces the model’s accuracy within that specific session.

3.3.2. Question-answer evaluation based on question-level collections

Each question was answered in forty separate test sessions, resulting in a set of forty responses per question. This allowed for a comprehensive tracking of each question’s performance across all test cycles. By analyzing these results, it was possible to identify which questions were consistently answered correctly and which were frequently misunderstood. This evaluation supported the classification of questions based on their difficulty and clarity, offering insights into how ChatGPT interprets and adapts its responses, especially when working with variations in question structure across different tests.

This method proved especially useful for assessing ChatGPT’s accuracy in responding to questions in the Sorani dialect. Equation 2 was used to calculate the truth ratio (TR), representing the proportion of correct answers for each question or test session. Following this, Equation 3 was applied to determine the proportion of incorrect responses (ER), providing a balanced view of the model’s strengths and limitations in handling Sorani language queries.

In this context, T represents the number of correct responses, also referred to as “True” answers. N denotes the total number of responses collected. These two values are essential for calculating the accuracy or TR of the model’s performance, as they allow for a straightforward comparison between the number of correct answers and the overall number of attempts.

In this evaluation, F represents the number of incorrect responses, also referred to as “False” answers. This value is used to measure the frequency of errors made by ChatGPT during the test rounds for each question. Question accuracy across all accounts is evaluated to determine how well each individual question performed throughout the test sessions. This is done using Equation 4, which calculates the accuracy of each question by dividing the number of correct responses by the total number of attempts. Specifically, T_j represents the number of correct responses to question j, and R_j denotes the total number of attempts for that same question. This measure provides insight into which questions were consistently understood and answered correctly by ChatGPT, helping to identify patterns in question difficulty and clarity.

4.1. Test Rate Evaluation

The comparison study conducted across the four test accounts, each exposed to ten repetitions per question, demonstrates a statistically significant variation in the consistency of response performance. These findings indicate the necessity for focused methodological interventions and model optimization tactics to enhance the overall dependability and accuracy of the system’s outputs.

Test results show that ChatGPT was more correct than incorrect answers, showing that this tool can be relied upon. Table 2 presents each account’s average number of true and false responses to 40 test questions, as well as each account’s ratio of true to false responses. As for the average of the tests, it is higher than the average of the true answers, being 34.875 and 0.6975, respectively, which indicates a strong tendency toward the correct answers. The lower average of 15.125 with a false response ratio of 0.3025, on the other hand, indicates false answers.

TABLE 2: The average response to all test questions

To show the direction of the TR and the ER, 40 tests were performed on a set of four accounts: A, B, C, and D; each of which was tested ten times over the course of 5 days, as outlined in the methods section. The TR starts relatively high at about 70% but then significantly drops between the first and the early tests where it approximates 55% in test 4. The ER, on the other hand, starts at around 30% and reaches its maximum of approximately 50% around there. Beginning with test 7, there is a dramatic turn in that the TR is significantly higher and remains consistent between 70% and 80% with the ER dropping low and stabilizing between 20% and 30%. The analysis maintains this correlation between the two ratios. The ratios are seen to stabilize during the tests, showing that the performance level is reached. The higher TR and lower ER, achieved posttest 7 in Fig. 4, indicate that the subject was overcoming some difficulties.

Fig. 4. The frequency of correct and incorrect responses across all tests.

Four accounts were compared in terms of their reaction to a series of test questions, and the results differed both in accuracy and the rate of errors. The analysis is based on the average number of true and false answers given by each account, along with the TR and the ER percentages, respectively. There is also an overall average for all four accounts to provide a point of comparison. A segmentation for this breakdown is presented in Table 3.

TABLE 3: The average responses to test questions for each account

The performance of four different accounts has been evaluated based on their responses to a series of test questions, revealing notable variations in accuracy and error rates. When examining the true answers, the average number of correct responses increases progressively across the accounts. Account_A demonstrates the lowest average with 30.3 correct answers, while Account_D achieves the highest with 37.6. This steady improvement indicates a gradual enhancement in performance from Account_A to Account_D. Conversely, the false answers show an inverse pattern: The average number of incorrect responses decreases from 19.7 for Account_A to 12.4 for Account_D. This inverse relationship between correct and incorrect answers supports the conclusion that Account_D performs the best, while Account_A leftovers the most.

Further evidence is provided by the TR and ER, which reinforce these observations. The TR, representing the percentage of correct answers, increases from 60.60% for Account_A to 75.20% for Account_A. Similarly, the ER decreases from 39.40% to 24.80% over the same range. These trends confirm the relative performance levels of each account and highlight the potential for targeted improvements. An overall average across all accounts is also included to serve as a baseline for comparison.

The performance of the four accounts has been tested by trial on ten consecutive average test questions and has been concerned with the ratio of correct to wrong answers. The comparison provides us with insight into how well each account performed over time and emphasizes fluctuation in accuracy. A clear visual representation of these trends is provided in Figs. 5 and 6. The primary purpose of these figures is to visually represent the fluctuations in the ratio of correct responses for each account, providing a comparative analysis of their performance over time. The proportion of correct responses is measured between 40.00% and 85.00%, and the testing axis indicates the time-wise arrangement of the ten tests. The performance of each account is plotted separately in a way that an explicit comparison and contrast could be shown for the test period.

Fig. 5. The ratio of correct responses based on account testing.

Fig. 6. The ratio of incorrect responses based on account testing.

Account_A exhibits the highest level of performance variability among the four accounts. It begins with an accuracy of approximately 70%, followed by a sharp decline to below 50% by the fourth test. This low level persists until the eighth test, after which a sudden rebound restores its performance to the initial range. Such fluctuations indicate a lack of stability in Account_A’s performance and may reflect underlying issues affecting its ability to generate accurate responses. In contrast, Account_B maintains a relatively stable accuracy throughout the testing period, fluctuating within a narrow range of 65% to 75%. Minor rises and dips are observed, but overall, the account demonstrates consistent performance with minimal variation, indicating reliability over time. In addition, Account_C also performs at a relatively high level, with accuracy predominantly exceeding 70%. Although it shows some moderate fluctuations, including a slight decline during the middle phase of testing, it recovers toward the end. This pattern suggests a generally resilient performance, with temporary setbacks that do not significantly impair overall accuracy. Finally, Account_D consistently achieves the highest performance among all accounts. Its accuracy remains above 70% across all tests, peaking at approximately 80% during the fourth test. While there are minor variations, its overall trend reflects a stable and robust level of accuracy, indicating dependable performance across the entire testing period.

These results highlight that each account produced different outcomes in response accuracy when evaluated using test questions in the Kurdish Sorani dialect through ChatGPT tools or different Chatbots. This fluctuation implies that AI performance is not uniform across accounts and may be influenced by training discrepancies or disparities in request zones. Consequently, these findings underline the need for continued refining of AI technologies to ensure uniform and equal performance across languages and user scenarios.

4.2. Questions Rate Evaluation

Nonetheless, the use of interpreters for every dialect significantly impacts the chatbot responses and is an essential factor in training and handling user inputs of multiple languages. As a result, the system occasionally gives different responses to a repeated question of which some might be correct and the others not correct. Inconsistencies typically occur when the dialog manager tries to give responses without enough information about the context of the question.

In particular, ChatGPT exhibits notable limitations in handling the Kurdish language, particularly in the context of Kurdish Sorani literature, poetry, and the knowledge of Kurdish poets. This subsection evaluates the response patterns from four labeled accounts, focusing on how their answers vary, sometimes randomly, when presented with specific questions related to Kurdish Sorani. The discrepancies point toward the reality that ChatGPT does not have adequate knowledge in such domains and therefore generates faulty outputs. Therefore, this analysis provides valuable insights and proposes future research for enhancing AI performance on underrepresented languages. It also requests enrichment of training data sets and language resources, especially for a dialect like Kurdish Sorani, to render future AI responses more accurate and culturally appropriate.

The discrepancies point toward the reality that ChatGPT does not have adequate knowledge in such domains and therefore generates faulty outputs. Therefore, this analysis provides valuable insights and proposes future research for enhancing AI performance on underrepresented languages. It also requests enrichment of training data sets and language resources, especially for a dialect like Kurdish Sorani, to render future AI responses more accurate and culturally appropriate.

Accordingly, the valuation of results highlights how frequently various feature test cases occurred across all accounts, providing a quantitative view of scenario distribution and performance levels. Test case identifies each scenario of counting the number of question-response (QR), frequency shows its occurrence count of responses according to the testes, and ratio reflects its percentage of the total. All summarized data are shown in Table 4. In the feature test, all questions were answered correctly in 18 tests, representing 36% and incorrectly in 3 tests, representing 6%. These three questions, which consistently receive incorrect answers, are related to poems, poet names, and historical knowledge. This significant discrepancy suggests either a high level of accuracy in other areas or a strong bias toward providing correct responses elsewhere.

TABLE 4: The frequency ratio of feature test cases across all accounts for each question.

Moreover, the subsequent segmentation of test cases is based on the number of correct answers within a defined threshold. In particular, cases where the number of correct answers ranged from 30 to 39 occurred 13 times, accounting for 26% of all tests. Although this result based on all questions is considered acceptable, the outcome when considering only the questions with all query-related responses marked as true reaches approximately 61%. This indicates that achieving optimal performance does not necessarily lead to a higher chance of correctly answering the selected query-related questions.

Test cases with a maximum of 20 correct ones were seen 4 times, and these comprised 8% of all tests. Cases with a maximum of 6 and a maximum of 0 correct ones each also appeared 6 times, representing 12%, respectively. These categories enable us to take a more general look at distribution in performance, with differing rates of accuracy across accounts. Notably, earning up to 30 accurate answers showed the second-highest frequency after full-score situations, validating the observation of generally excellent accuracy. The results connected to QRs provide insight into performance trends and accuracy across tests, demonstrating the overall reliability of the tested accounts. However, many restrictions remain, particularly in generating accurate replies in Sorani Kurdish. This reflects limitations in ChatGPT’s handling of some Kurdish dialects and domain-specific information, especially when striving for optimal answers across multiple user accounts and geographies.

One of the cases in the study illustrates the TR and ER for all questions across all accounts, based on the frequency of correct and incorrect QR responses over a set of 50 questions. The primary objective is to visually demonstrate the variation in the percentage of correct and incorrect answers across this range, as shown in Fig. 7.

Fig. 7. The rate of correct and incorrect responses for all questions across all accounts.

It can be observed that with high TR, ER will be low, and low TR will result in high ER, which models an anticipated reverse relationship between correct and erroneous QR. The measures will normally spike to 100%, which indicates complete accuracy, yet fall to 0%, which indicates complete inaccuracy. Such extreme fluctuation reflects significant variation in performance, featuring normal swings between absolute accuracy and absolute error, consequently implying erratic response quality between accounts.

The analysis provides a comparison of feature test cases in four accounts. The main intention is to describe the distribution of each account’s test result, including information about their relative performance and stability. It was found that QR frequencies may differ between accounts, with time zone and domain being potential deciding factors. Thus, the division is by the frequency at which each test case appeared by account or by the frequency at which each given result occurred. These results are summarized in Table 5.

TABLE 5: The frequency ratio of feature test cases across different accounts

The findings categorized the cases according to the total number of correct answers to QRs per account. Account_B has the highest count at 30 correct answers out of a total of 50 questions answered correctly, followed closely by Account_C at 28 and Account_D at 27 correct answers. Account_A has a relatively lower frequency of 18. For cases of all questions being answered wrongly, Account_B leads with 7 while Account_A indicates 6 and Account_C and Account_D each indicate 5. This implies that while Account_B is the one with the highest number of perfectly correct answers, it also indicates a relatively higher rate of overall failure among the accounts. In addition, Table 5 segment splits tests into various groups based on the number of QRs in given intervals. Account_D has at most 5 correct test scores with 9; Account_A and Account_C have 8 tied, and Account_B has 3. Account_A and Account_C have <5 correct test scores with 10 and 9, respectively, Account_B has 9, and Account_D has 5, meaning Account_D with fewer low scores. For more precise 5 correct answers for examinations, the top frequency is in Account_A with 8, then Account_D with 4, Account_B with 1, and Account_C with zero, indicating high heterogeneity across accounts.

The frequency of correct responses across four different accounts over a series of 50 questions is illustrated in Fig. 8. The graph provides a visual comparison of how the number of accurate answers varies for each account, highlighting differences in performance throughout the set. Conversely, Fig. 9 displays the opposite of Fig. 8, indicating the frequency of wrong replies across the QRs for the four different accounts. The frequency reflects the number of correct or incorrect responses, ranging from 0 to 10 tests per account over 50 distinct questions. Each account’s performance is represented independently, allowing for easy separation and comparison throughout the question sequence.

Fig. 8. The frequency of truth responses for each different account.

Fig. 9. The frequency of false responses for each different account.

The response patterns of the four accounts are very different in consistency and response correctness. Account_A is the most variable with large oscillations between 0 and 10 correct responses, which show extreme inconsistency although its correct responses show fewer errors than others. Account_B is also fluctuating but less wildly and in a pattern that indicates a slightly more stable but still fluctuating performance. Account_C has only moderate volatility, with most times having a higher proportion of correct responses with fewer instances of zero, indicating more stable accuracy. Account_D is characterized by the most stable and consistently high level of performance, staying very close to 10 correct responses on all accounts, indicating solid and consistent ability.

The results clearly show that the interpretation of queries by chatbots using ChatGPT varies significantly, particularly when different dialects are involved. This variation is especially evident in the test queries analyzed during this investigation. The accuracy and consistency of responses differed depending on the dialect used, highlighting challenges in understanding and processing language variations. In addition, external factors like the time zone that the test was aimed at and response time per question also contributed. These could have skewed the performance of the chatbot, leading to variations in results between accounts.