Comparative Analysis of Word Embeddings for Multiclass Cyberbullying Detection

1. INTRODUCTION

Cyberbullying involves the consistent use of electronic or digital platforms by individuals or collectives to repeatedly convey harmful or aggressive messages with the intent of causing discomfort or harm to others [1]. This encompasses actions such as sending cruel or menacing messages, sharing humiliating photos or videos, propagating gossip, or crafting fictitious profiles or websites with the purpose of shaming an individual. Cyberbullying can lead to significant repercussions for both the target and the person engaging in it. Victims of cyberbullying may experience a range of negative effects, including feelings of sadness, anxiety, depression, and even thoughts of suicide. They may also have trouble sleeping, lose interest in activities they used to enjoy, and struggle with self-esteem [2]. Individuals engaging in cyberbullying might face serious repercussions, including disciplinary action from educational institutions or legal authorities, as well as potential legal consequences. Such actions could also negatively impact their future relationships and life opportunities. It is crucial to understand that cyberbullying is not confined to a specific age group or demographic; it can affect anyone. Although young people are commonly involved, adults are not immune to being targeted by cyberbullying [3].

Automatic cyberbullying detection refers to the use of computational methods and technologies to automatically identify instances of cyberbullying in electronic communications, such as text, images, and videos.

This typically involves the utilization of natural language processing (NLP) techniques, machine learning (ML) algorithms, and other analytical approaches to analyze the content and context of electronic communications [4]. Different techniques have been put forth for automated cyberbullying identification, encompassing rule-based models, conventional ML models, and deep learning models [5]. Rule-based models depend on pre-established rule sets and lexical resources to detect instances of cyberbullying. On the other hand, conventional ML models employ algorithms such as logistic regression, decision trees, K-nearest Neighbors (KNN), and support vector machines to categorize text as either cyberbullying or not. Deep learning models, conversely, use neural networks to identify patterns in the data that indicate cyberbullying. Research on automatic cyberbullying detection has mainly focused on social media platforms such as Twitter, Instagram, and YouTube [5]. These platforms are particularly relevant for cyberbullying detection due to their popularity among young people and the large amount of user-generated content that is available for analysis.

Research in the literature has used the following features to improve the accuracy of detection [6, p. 2].

2. RELATED WORK

In their research, Yin et al. [8] have the first to explore automated recognition of cyberbullying in online environments. They collected three sets of data from three distinct online platforms to detect instances of harassment. One dataset was sourced from the Kongregate platform, whereas the other two were collected from Reddit. To classify the data, the authors used a linear kernel classification model and applied different techniques for feature extraction, including N-grams and term frequency-inverse document frequency (TF-IDF). Despite having uncertain experimental results, the study was a significant step toward further research in this field.

Researchers in Iwendi et al. [9] explored the effectiveness of four deep learning models – BLSTM, gated recurrent units (GRU), long short-term memory (LSTM), and recurrent neural network (RNN) – in detecting cyberbullying. They applied rigorous data pre-processing steps, including text cleaning, tokenization, stemming, lemmatization, and removal of stop words, before feeding the data into the models for prediction. Among these models, BLSTM stood out, showing higher accuracy and F1-measure scores compared to RNN, LSTM, and GRU [10] proposed a ML approach for detecting cyberbullying in text-based online communication. The authors collected a large dataset of messages from social networking sites and labeled them as either bullying or non-bullying. Subsequently, they derived a range of linguistic attributes from the messages, encompassing syntactic and semantic aspects, alongside social network characteristics. The authors employed various ML techniques, such as decision trees and support vector machines (SVMs), to categorize messages as either exhibiting bullying behavior or not.

Bozyiğit et al. [11] collected a dataset from twitter for Turkish language and used social features to detect cyberbullying, the authors also created a web application that detects cyberbullying live. In Aizawa [12], an approach based on feature engineering was introduced for identifying multi-class cyberbullying on Twitter. The authors utilized a dataset containing 10,000 tweets that were labeled with three levels of cyberbullying severity: low, medium, and high. Initially, they employed the synthetic minority oversampling technique to oversample the underrepresented categories (low, medium, and high) by a factor of 300. Subsequently, they applied a weighted cost for misclassification in the minority categories. On the other hand, the study outlined in Dinakar et al. [13] utilized deep learning architectures to investigate the performance of various deep learning algorithms (LSTM, BiLSTM, RNN, and GRU) in terms of their effectiveness for identifying antisocial behavior. The researchers performed empirical assessments to measure how well the algorithms work in recognizing antisocial behavior. [14] utilized a dataset collected from twitter with 16K records, they applied TF-IDF, task-specific embeddings to detect hate speech that was categorized to sexist, racist, or none comments, the authors reported an F1-score of 0.93.

In Agrawal and Awekar [15], the authors used YouTube comments data that they used TF-IDF and a collection of profane words then applied Naïve bayes, SVM, J48, and JRip to detect three types of cyberbullying, namely, sexuality, race, and intelligence that they achieved a maximum accuracy of 80.20% on sexuality cyberbullying. [16] used char-n grams, word n-grams as features on Wikipedia dataset that they applied Logistic Regression and Multilayer Perceptron to categorize whether the comments included personal attacks or not.

In Wulczyn et al. [17], the authors examined various content characteristics, such as the use of first- and second-person pronouns, as well as the presence of vulgar language. They found that these factors indeed served as markers for cyberbullying.

The study [18] involves the utilization of a deep learning algorithm in conjunction with fuzzy logic to analyze 47,733 comments from Twitter, obtained from Kaggle. The methodology includes processing these comments using Keras embeddings and classifying them with a four-layer LSTM network. The application of fuzzy logic then aids in determining the severity of the flagged cyberbullying comments.

A comparable method was employed in Agrawal and Awekar [15], where researchers mainly focused on content attributes such as racist language and profanity. Another interdisciplinary study, discussed in Mikolov et al. [19], approached the issue from both computer science and human behavior perspectives. In their proposed technique, the researchers extensively analyzed content features, including URLs and hashtags. Surprisingly, they observed that 64 of the tweets contained external links, and 74.2% included hashtags, but these two aspects were not indicative of bullying. Similarly, [20] continued to explore cyberbullying detection through a content-based lens, introducing new elements like emoticons and a hieroglyph dictionary. They evaluated their approach using various ML algorithms, including SVM and J48, with SVM achieving the highest accuracy rate of 81%.

Another approach in the literature about tackling cyberbullying issue has been using transformer-based models such as BERT [21], BERT-m [21], [22], DistilmBERT, and IndicBERT [23]. Hybridizing various deep learning models, including Simple RNN [14], LSTM, BiLSTM [14], GRU [24], and CNN [25].

Dinakar et al. [13] Deep neural networks were utilized on three different datasets (Formspring, Twitter, Wikipedia), the authors in Badjatiya et al. [16] used basic word embeddings and language models including RoBERTa, XLNET, and ALBERT to detect cyberbullying. They evaluated their approach on a dataset of 10,000 tweets and found that the best-performing word embedding method is RoBERTa, which achieves an accuracy of 93.2%.

3. MATERIALS AND METHODS

Fig. 1 shows the proposed methodology of this research; in the following subsections, we discuss each part of the methodology.

Fig. 1. Proposed methodology.

3.1. The Dataset

The dataset has been prepared by [26] that contain 47,692 records distributed approximately equally amongst six classes as shown in figure 2, namely, not cyberbullying, religion, gender, ethnicity, age, and other. The dataset has been collected from the twitter social network. The dataset offers the opportunity to establish a multiclassification model for forecasting cyberbullying types, develop a binary classification model to identify potentially harmful tweets, or investigate the words and patterns linked with each form of cyberbullying. Fig. 1 shows sample distribution per type.

Fig. 2. Sample distribution amongst classes.

3.2. Text Preprocessing

In the preprocessing phase, the following actions shown in Fig. 3 were performed to clean and transform the text before further analysis or modeling.

Fig. 3. Preprocessing steps taken.

3.2.1. Remove links

This is the initial cleaning step where any hyperlinks contained in the text are removed. Links may not contribute to the analysis of the text’s content, especially when the goal is to understand linguistic patterns or sentiment.

3.2.2. Clean non-alphabetical characters

At this stage, the text is further cleaned by removing any characters that are not part of the alphabet. This typically includes punctuation, numerical digits, symbols, and any other non-letter characters. This step helps in focusing the analysis on words only.

3.2.3. Convert text to lowercase

Converting all text to lowercase is a standard normalization technique that helps in maintaining consistency throughout the dataset. It ensures that the same word in different cases (e.g., “Word” vs. “word”) is treated as identical during analysis.

3.2.4. Tokenize

Tokenization is the process of splitting the text into individual terms or words. This is a crucial step because most language processing methods work at the word level.

3.2.5. Remove stop words

Common words such as “the,” “is,” and “in,” which appear frequently and have little lexical content, are removed in this step. By eliminating these, the data are distilled, focusing on the more meaningful content words.

3.2.6. Stemming

Stemming algorithms work by cutting off the end or the beginning of the word, considering a list of common prefixes and suffixes that can be found in an inflected word. This process reduces words to their root form, which helps in generalizing different forms of the same word (e.g., “running,” “ran,” “runs” all become “run”).

Although text preprocessing steps can impact the original meaning of tweets to some extent yet their importance in reducing noise and shifting focus on the most meaningful words remains crucial. While stop words (such as “the” and “is”) are often grammatically essential, they usually carry little semantic weight. The subsequent word embedding stage effectively captures the deeper semantic and syntactic relationships between the remaining words. Therefore, although some nuances may be lost, this trade-off is a necessary part of optimizing the model for efficiently detecting patterns indicative of cyberbullying.

3.4. Classification (CNN-BiLSTM)

Following the embedding layer is a bidirectional layer encapsulating an LSTM layer which is depicted in Fig. 4. The LSTM is a type of RNN designed for sequential data processing. The bidirectional wrapper enables the LSTM to process input in the forward and backward directions simultaneously. Moreover, the LSTM layer is configured to return the full sequence.

Fig. 4. Convolutional neural networks-bidirectional long short-term memory model.

Subsequently, a convolution1D layer is added, primarily used for one-dimensional convolution applied to sequential data, such as text. The layer requires three parameters: The number of filters (32), the kernel size (8), and the activation function employed (ReLU). To perform global max pooling on temporal data, the model incorporates a GlobalMaxPool1D layer, thereby extracting the maximum value across all time steps. To flatten the input, a Flatten layer is introduced, which reshapes the multidimensional output into a one-dimensional vector.

Next, a Dense layer with 128 neurons and a ReLU activation function is appended. A dense layer represents a regular layer of neurons within the neural network, with each neuron receiving input from all neurons in the preceding layer, establishing dense connections.

To combat overfitting, a Dropout layer with a dropout rate of 0.2 is included. Dropout serves as a regularization technique by randomly deactivating a fraction of input units during the forward pass. The final layer added is a dense layer consisting of 6 neurons and employing a SoftMax activation function. This layer is tasked with producing the probabilities linked to the input’s membership within each of the six classes. The model is ultimately compiled with the sparse categorical cross-entropy loss function, the Adamax optimizer, and accuracy as the evaluation metric.

4. RESULTS AND DISCUSSION

This section presents the findings of our experiments, where we utilized deep learning models to create a cyberbullying detection system (CDS). The purpose of the CDS was to detect and categorize instances of linguistic cyberbullying into multiple classes. Regarding the choice of the word embeddings used in our experiments, pre-trained word embeddings performed better than trainable word embeddings.

In this investigation, we assessed the efficacy of a proposed model by utilizing various evaluation metrics to gauge its capacity for distinguishing the six classes of cyberbullying. The following commonly used evaluation criteria are employed to gauge the effectiveness of cyberbullying classifiers for social media networks:

• Accuracy (equation 1): This measures the ratio of correctly identified instances to all cases and is frequently employed to evaluate the performance of cyberbullying prediction models.

• Precision (equation 2): Precision determines the percentage of relevant tweets among tweets classified as both true positives and false positives for a given category.

• Recall (equation 3): This metric assesses the proportion of relevant tweets that are correctly identified out of all relevant tweets.

• F-measure (equation 4): The F-measure offers a unified metric that considers both recall and precision, providing a balanced assessment of both aspects.

However, since the dataset is balanced these metrics produced similar numbers; therefore, we report the accuracy of each embedding in (Fig. 5). The bar chart visualization of the results showed that fine-tuning BERT achieved the highest accuracy of 85%, indicating its effectiveness in identifying instances of cyberbullying. Glove and Word2Vec followed closely behind with accuracies of 82% and 81.31%, respectively, showcasing their strong performance as well. The Trainable model, which represents a non-trained approach to word embeddings, achieved an accuracy of 80.2%. We used a batch size of 32 and ran the model for six epochs.

Fig. 5. Accuracy of different word embeddings for the convolutional neural networks-bidirectional long short-term memory model.

Regarding the individual classes of cyberbullying, the two classes of other cyberbullying and non-cyberbullying recorded worse results in all the experiments (Fig. 6) show the accuracy, precision, recall, and F1-score of BERT model. In an attempt to further increase model performance, [30] have used a modified version of this dataset where the other cyberbullying class has been removed from the dataset.

Fig. 6. Bidirectional encoder representations from transformers metrics for the six classes of the dataset.

To further investigate the results achieved with our proposed method, we conducted tests using several classical ML algorithms, as shown in Fig. 7: Logistic Regression, KNN, Random Forest, and Naïve Bayes. The data underwent the same preprocessing steps, as described in Fig. 3. Subsequently, we applied the TF-IDF vectorizer to extract features and reduce the dimensionality of the vocabulary.

Fig. 7. Accuracy of baseline models.

TF-IDF stands as a numerical metric utilized in information retrieval and text mining. Its purpose is to gauge the significance of a term within a specific document or an entire collection [12]. For our experiments, we allocated 20% of the dataset for testing and used the remaining 80% for training.

For logistic regression, we set the maximum iteration to 300 and chose lbfgs as the solver. As for KNN, we selected k=3 and used uniform weights. In the case of random forest, we specified a maximum depth of 100. All other hyperparameters were set to the default values provided by the sklearn library [31].

5. CONCLUSION AND FUTURE WORK

This research article focuses on the application of word embeddings for multiclass cyberbullying detection. The study compares the performance of trainable word embeddings, pre-trained word embeddings, and fine-tuning language models using various techniques and models such as CNN and biLSTM, as well as word2vec and BERT.

The research goes beyond simple binary classification and instead focuses on multi-class detection of cyberbullying. By employing a dataset collected from Twitter, the study explores the effectiveness of different word embedding techniques in detecting cyberbullying types such as religion, gender, ethnicity, age, and others. The preprocessing phase includes actions such as removing links, cleaning text, tokenization, removing stop words, and stemming.

The results indicate that fine-tuning the BERT model on the dataset yields the most promising results. The proposed CNN-BiLSTM model, incorporating word embeddings and deep learning techniques, demonstrates effectiveness in classifying and detecting cyberbullying instances.

In the future, it is important to focus on domain adaptation and transfer learning for cyberbullying detection. Researchers should investigate techniques for adapting models to different social media platforms or domains. Exploring transfer learning approaches that leverage knowledge from related tasks or domains can enhance the performance of cyberbullying detection models. Creating cross-domain datasets will enable the evaluation of domain adaptation and transfer learning techniques in realistic settings. In addition, developing knowledge distillation methods can facilitate the transfer of knowledge to smaller, more efficient models. Real-world evaluations are crucial to measure the effectiveness of domain adaptation and transfer learning techniques, considering variations in data distribution, biases, and user behaviors. By enhancing the generalizability of cyberbullying detection models, we can better address the challenges across diverse platforms and user contexts.

The findings can aid in the development of more accurate and efficient methods for identifying and preventing cyberbullying, particularly in the context of social media platforms.