Optimization of Lattice-Based Cryptographic Key Generation using Genetic Algorithms for Post-Quantum Security

1. INTRODUCTION

The launch of quantum computing has considerably challenged the standard cryptographic setups [1]. Existing classical cryptographic systems, such as RSA and Elliptic Curve Cryptography (ECC) [2], bank on mathematical complexities that are computationally very hard for standard computers but can be quite simply solved by quantum algorithms, such as Shor’s algorithm [3]. This weakness has sped up the search for post-quantum computing (PQC) solutions and eventually led to the design of cryptosystems that are amenable to attack by quantum computers [4]. Lattice-based-cryptography (LBC) [5], with its robust security guarantees and great ease of computation has since been in the limelight. In this spirit, the National Institute of Standards and Technology (NIST) [6] has broadened its interests in forming global efforts to standardize PQC algorithms: Some, like lattices, are under scrutiny to become future standards for cryptographic schemes.

Lattice-based cryptography (LBC) is based on the extreme hardness of two computationally challenging lattice problems called the Shortest Vector Problem (SVP) and the Learning with Errors (LWE) [7]. This kind of cryptography is considered a potent building block for digital signatures, key agreements, and encryption security for the quantum age [8]. However, the practical deployment of such systems faces challenges in key generation and optimization. Key sizes in large key lattice systems may yield troublesome memory sizing. At the same time, computational cost may restrict their applicability in low-resource environments such as the Internet of Things (IoT) and embedded systems [9].

LBC derives its security from the computational intractability of certain fundamental lattice problems, primarily the SVP and the LWE problem [7]. The SVP involves finding the shortest non-zero vector in a high-dimensional lattice, which is known to be NP-hard under specific norms [10]. The LWE problem, on the other hand, is based on solving linear equations with small random errors, making it resistant to both classical and quantum attacks. These problems serve as the basis for constructing secure encryption schemes, digital signatures, and key exchange protocols [11]. Unlike factorization-based cryptosystems, which are vulnerable to Shor’s algorithm, the hardness of SVP and LWE remains intact even in the presence of quantum computers [12]. As a result, LBC has emerged as a leading candidate for PQC and is actively being standardized by NIST for future cryptographic applications.

To tackle these problems, genetic algorithms (GAs) are used in this study to address the optimization incorporating lattice-based cryptographic keys. GAs [13], which were taken from biology around 1970, are an efficient and effective technique for solving optimization problems in complex algebraic and other structures. GAs have, in so many words, implemented all the desirable features that were essential to hyperplanes’ construction [14], such as characters, crossover/mutation, strategies, concealing, and non-linearities. Through the practice of GA, it is possible to refine lattice-based cryptographic keys by adopting an evolutionary multi-objective technique selection, crossover, and mutation. The GA and LBC concepts meshing enhance the key generation process by pruning the infeasible keyword combinations that do not guarantee high security.

Major Contributions:

The remaining sections of this paper are as follows: Section 2 provides a comprehensive survey of PQC and lattice-based schemes and explanatory notes on optimization. Section 3 shows how GAs are combined with lattice-based key generation and explains the research strategy used. Section 4 describes an experimental study on the improvement method, including the performance measurement and the results obtained. Finally, Section 5 provides closure to the text and delves into the prospective directions of the study’s development.

2. LITERATURE REVIEW

A serious threat to traditional cryptographic schemes by quantum computers has spurred the development of PQC – an arms race for creating cryptographic algorithms capable of withstanding quantum-level attacks. The quest for more robust encryption methods in the future of quantum computing has led the NIST to assume a leading role in standardizing such algorithms. It then seeks an extensive account of recent pasta and relevant methodologies of PQC in LBC, its applications, improved performance, and enhanced security against threats from quantum computers.

Quantum computers use quantum mechanical principles to perform tasks at exponentially faster speeds than classical ones. Notably, integer factorization, which is an example of the task, could be broken through the use of Shor’s algorithm, thus making the most used cryptographic systems, such as RSA and ECC “dead.” Although Shor’s algorithm has only been deduced by theory, it is powerful or efficient enough to continue with any practical quantum computer for big-sized everyday applications. Thus, the emergence of PQC would keep it secure even against the quantum era. Standardization about resistance to quantum algorithms is yet crucial, and even candidates have been evaluated at NIST primarily based on complex mathematical problems, e.g., LBC [15].

The rapid advancements in quantum computing pose significant threats to traditional cryptographic systems, necessitating a transition toward PQC. UAVs, which rely on open wireless channels and have constrained computational resources, are particularly vulnerable to quantum-based attacks, making PQC integration crucial for securing their communications [16]. The NIST PQC standardization process has driven research toward hardware implementations, leading to the development of PQC hardware circuits and system design techniques to enhance security in various applications [17]. However, the migration of IP networks to PQC remains a complex challenge due to inconsistencies in migration steps, terminology, and security concerns, with present implementations being largely experimental [18]. A structured transition plan to PQC is essential, incorporating identification, protection, detection, and response phases to ensure a quantum-resilient cybersecurity infrastructure [19]. While hybrid cryptographic systems combining traditional and quantum-resistant techniques are being explored, significant challenges remain in practical deployment and regulatory alignment.

One of the candidates in the post-quantum computing (PQC) contest, LBC, has become a beautiful choice. LBC entails the solving of lattice-based problems such as the closest vector problem, the flexible LWE, and the ring LWE issues that are known to be classically and quantumly hard to solve [20]. LBC-based protocols are available for encryption, key exchange, digital signature, and even homomorphic encryption, and they can be used to secure various kinds of data and communication channels. Even though LBC is safe in practice, some real-world problems arise while trying to implement this scheme, where primary key and memory management structures may be inefficient, making it not convenient for use in areas where resources, such as embedded systems and IoT equipment, are few.

The global spread of IoT devices and the subsequent advancement in capabilities of mobile networks to 5G has necessitated the search and development of cryptosystems that will resist attacks facilitated by a quantum computer. IoT devices are of limited computational power, which requires lightweight cryptographic solutions that provide resistance to quantum attacks and are not stretched when it comes to performance [21]. Such a lightweight LBC instance is beneficial in such a scenario as it provides almost the same level of security as the ordinary LBC with approaches that decrease both memory usage and the level of computing resources in general [22]. In particular, a recent study provided efficient symmetric and asymmetric quantum-safe algorithms for an IoT environment, showing significant performance and memory cost improvements over the existing NIST-approved algorithms [21].

With its abundance of highly complex operations, the prospect of LBC has put a lot of pressure on its developers to improve performance [23]. Various forms of vectorization have been instrumental in enhancing lattice-based schemes, namely key exchange protocols. The section has improved and reduced time and energy consumption [24]. Finally, attention was focused on the problems associated with enlarging keys and using lattice-based systems. An example of such systems is NTRU systems, which can provide faster encryption and key generation than RSA but also suffer from the problem of the large sizes of the keys, which renders them unsuitable for many users [25].

LBC has the potential to address challenging security issues, but great care is still required to protect all such systems from the effects of quantum computers. There continue to be hazards associated with lattice structures and their operations, irrespective of whether they are hardware or software components. For that reason, research in this area is necessary to improve the security for LBC and come up with workable measures in the wake of new threats, such as those that may rely on stealth, as seen in side channel methodologies or physical access [26]. Moreover, the transformation of NIST’s PQC faceless approach has to consider the security necessity along with the practicality of memory consumed, scheduler cycles utilized, and power consumed by the devices to ensure that the said tablets satisfy the limits of different hardware equipment [27].

Cryptographic systems that are lattice-based have arisen as the answer to the problem of post-quantum vulnerability. This security concern is determined by the specific algorithms encountered while encrypting information. Although the security model of such systems is rather strong, certain complexity is observed in their practical implementation in environments where resources are limited, such as the IOTs or Edge processing. Optimizing LBC production or incorporating other LBC optimal techniques, such as vectorization, has dramatically increased the ability of LBC to adapt to different appliances. Moreover, it may be expedient to evaluate the security, performance, and other parameters in scale as pioneers of this technology accept other changes that are inevitable toward quantum computers. NIST’s attempts to invent Projected Quantum Computation Standalone systems for the future are significant for the following safety standards needed. A comparison of the most relevant related works is provided in Table 1 to highlight the differences and advantages of the proposed method.

TABLE 1: Comparison of the related works

3. MATERIALS AND METHODS

3.2. GA

GA [13] is an evolutionary optimization technique inspired by natural selection and genetics principles. It is widely employed to solve complex optimization problems where traditional mathematical or heuristic approaches may be inefficient. GA operates on a population of candidate solutions, evolving them iteratively through selection, crossover, and mutation operations to enhance solution quality. The fitness function, denoted as f(x), evaluates the suitability of each candidate solution x within the search space.

The algorithm begins with an initial population P = {x₁, x₂,…,x_n}, where each x_i is a chromosome encoding a potential solution. Selection is performed to retain high-fitness individuals, commonly using roulette wheel selection, tournament selection, or rank-based selection. In roulette wheel selection, the probability of selecting an individual x_i is proportional to its fitness:

Crossover, or recombination, involves exchanging genetic material between selected parents to create offspring. A common approach is single-point crossover, where a random crossover point c is chosen, and genes are swapped between two parents to produce new chromosomes:

Child₁ = (Parent₁[: c], Parent₂ [c: ]) (3)

Child₂ = (Parent₂[: c], Parent₁ [c: ]) (4)

Mutation introduces small random alterations to maintain genetic diversity and prevent pre-mature convergence. If x_i is a binary-encoded chromosome, a bit-flip mutation at position j can be expressed as:

x_ij = 1-x_ij (5)

Where x_ij is the j^th gene of chromosome x_i. The algorithm iterates through these operations until a termination criterion is met, such as achieving a pre-defined fitness threshold or reaching a maximum number of generations.

3.3. Proposed Method

This study presents a hybrid cryptographic framework hybrids GA and lattice-based encryption to enhance security and adaptability. The method consists of four main stages: key optimization, encryption, mutation, and decryption. Key optimization is performed using GA to generate an invertible encryption key matrix K over Z₂₅₆. The fitness function ensures the invertibility of K by verifying that its determinant satisfies gcd(det(K),256) = 1 and that its modular inverse exists in modulo 256. The fitness function is defined as:

Where ∑K represents the sum of all elements in the key matrix. The GA iteratively refines K using selection, crossover, and mutation to obtain an optimal encryption key. Encryption transforms the plaintext message P into its numerical representation and computes the ciphertext C using matrix multiplication in modular arithmetic:

C = P⋅K mod 256 (7)

Where padding is applied to align with encryption parameters. To enhance security, a GA-based mutation process introduces controlled randomness by flipping bits in C, further obfuscating the encrypted text and increasing resistance against attacks. Decryption is performed using the modular inverse of the key matrix, ensuring correct retrieval of the original message:

P’ = C⋅K⁻¹ mod 256 (8)

Where P’ ideally reconstructs the original plaintext. The invertibility of K ensures accurate recovery of P without information loss. Integrating GA-based optimization with lattice cryptography provides a robust mechanism resistant to conventional attacks. The introduced noise through mutation adds a layer of obfuscation. The proposed scheme offers a potential post-quantum alternative, leveraging the hardness of modular matrix inversion while maintaining computational efficiency.

To address the potential security vulnerabilities associated with using GAs for cryptographic key generation, we acknowledge that while GAs offer efficiency improvements, they may introduce risks. Specifically, the optimization process might lead to patterns or predictable sequences in the generated keys, which could be exploited by attackers. This concern arises because GAs are inherently stochastic, and if not carefully designed, they may converge to weak or vulnerable keys, thereby reducing the security of the encryption system. In addition, if the search space is not sufficiently large or diverse, or if the genetic operators (such as selection, crossover, and mutation) are not effectively applied, the randomness required for secure cryptographic keys could be compromised, making them more susceptible to cryptanalysis techniques such as brute-force or statistical attacks.

To mitigate these risks, we have incorporated mechanisms within our GA-based key generation process to preserve randomness and avoid pre-mature convergence. This includes ensuring that the search space remains large and diverse, and that proper randomization techniques are used. We also recommend conducting further security evaluations, such as testing against known cryptographic attacks (e.g., differential and linear cryptanalysis) and assessing the strength of GA-generated keys in realistic environments. While GAs can enhance the efficiency of cryptographic key generation, it is essential to carefully design and test the approach to prevent the introduction of vulnerabilities and ensure its applicability in secure cryptographic systems. The flowchart of the complete process is illustrated in Figure 1, providing a visual representation of the step-by-step workflow.

Fig. 1. Flowchart of the proposed method.

4. EXPERIMENTAL EVALUATION

The experimental evaluation of the proposed method was conducted with a population size of 50 and a maximum iteration count of 50. The implementation was implemented in Python and executed on a Core i7-13650HX processor with 24GB of RAM. To optimize the efficiency of the GA in cryptographic key generation, we used the face-centered central composite design method [29], which tests parameters such as population size, mutation rate, and number of generations at three levels: low, medium, and high. This systematic approach allows for a thorough exploration of the parameter space to identify optimal settings. The following tables and figures comprehensively assess the method’s performance in encryption, decryption, and mutation processes.

Table 2 presents the optimized key matrices derived from the proposed method. The optimized key structure is critical to encryption security, ensuring randomness and unpredictability while maintaining computational efficiency. The displayed key values highlight the capability of the GA in optimizing cryptographic parameters. The key matrix exhibits a well-distributed pattern, reinforcing the robustness of the encryption scheme. These optimized keys are instrumental in generating secure ciphertext, thereby preventing unauthorized decryption attempts. The structured yet dynamic nature of the key suggests that the GA is effective in producing high-entropy keys, which is crucial for strong encryption. The results confirm that the proposed approach generates optimized encryption keys that enhance security and balance complexity and computational feasibility. By ensuring a high degree of randomness, the key optimization process significantly improves the overall encryption robustness, making it resistant to cryptanalysis.

TABLE 2: Optimized Keys achieved by the proposed method

Fig. 2 illustrates the convergence behavior of the GA in optimizing the key generation process for lattice-based cryptographic schemes. The x-axis represents the number of generations, while the y-axis denotes the best fitness value observed during the optimization process. As the number of generations increases, the fitness value improves consistently, demonstrating the effectiveness of the GA in evolving optimal cryptographic keys. The initial fluctuations in the fitness values indicate the exploration phase of the algorithm, where diverse candidate solutions are evaluated. As the generations progress, the curve exhibits a steady upward trend, reflecting the exploitation phase where the algorithm refines solutions to achieve higher fitness values. The smooth yet incremental improvement in fitness confirms the stability of the GA and its ability to converge toward an optimized key structure. This result validates the efficacy of using GA-based optimization in post-quantum cryptographic key generation by reducing computational overhead while maintaining high-security standards.

Fig. 2. The convergence curve is attained by the proposed method.

Table 3 presents the encryption and mutation results for short ciphertexts containing between 1 and 100 characters. The original plaintext and its corresponding decrypted output confirm the accuracy of the GA-based key recovery. The exact match between the decrypted text and the original message indicates that the proposed decryption approach successfully reconstructs the original input without introducing errors. Furthermore, the analysis of original and mutated ciphertexts reveals subtle alterations applied to the encoded text, a critical feature in enhancing security. These controlled mutations introduce diversity in the ciphertext representation, ensuring that even minor variations in the key can lead to significantly different encrypted outputs. This characteristic is essential in preventing pattern-based cryptanalysis while maintaining the ability to decrypt the message when the correct key is used accurately. The results validate the proposed approach’s effectiveness in ensuring secure communication with a robust decryption mechanism.

TABLE 3: Short ciphertext and mutation analysis 1–100 characters

Table 4 extends the ciphertext analysis to medium-length messages, ranging from 101 to 200 characters. Similar to the previous case, the decrypted text precisely matches the original, further establishing the reliability of the GA-based decryption mechanism. The analysis of the ciphertext mutations reveals systematic modifications applied to the encrypted message. These modifications serve the dual purpose of enhancing security while preserving the integrity of the decryption process. By introducing slight perturbations in the ciphertext structure, the mutation process ensures that even if an attacker gains partial knowledge of the encryption method, reconstructing the exact plaintext without the key remains computationally infeasible. The results highlight the robustness of the proposed method in handling larger messages while maintaining high decryption accuracy. This adaptability to varying message lengths is crucial to practical cryptographic implementations, making the method viable for real-world applications requiring secure communication.

TABLE 4: Medium-length ciphertext and mutation evolution 101–200 characters

Table 5 evaluates the performance of the proposed approach on long ciphertexts containing between 201 and 300 characters. The results reinforce the algorithm’s ability to decrypt lengthy messages accurately. The decrypted output remains consistent with the original, demonstrating the method’s scalability and reliability. The analysis of the mutated ciphertexts further substantiates the security-enhancing capabilities of the proposed approach. The systematic variations introduced in the encrypted text ensure that even slight modifications to the input lead to substantial changes in the ciphertext, a desirable property in cryptographic systems. These controlled mutations make it challenging for adversaries to exploit patterns or predict key variations, strengthening encryption security. The successful decryption of long messages while preserving security properties establishes the proposed method as a robust solution for high-volume data encryption scenarios, such as secure messaging and data storage.

TABLE 5: Long Ciphertext with GA-based decryption and mutations 201–300 characters

Table 6 presents another analysis of long ciphertexts, reaffirming the consistency of the proposed method in handling extensive encryption and decryption tasks. The decrypted output aligns perfectly with the original message, confirming the reliability of the GA-based approach. The mutations applied to the ciphertext exhibit controlled randomness, ensuring that each encryption process yields a unique representation of the plaintext. This diversity in ciphertext generation enhances security by preventing pattern recognition and statistical attacks. The ability to maintain decryption accuracy despite introducing systematic variations in the ciphertext underlines the robustness of the encryption framework. These findings further validate the scalability and efficiency of the proposed approach, making it a viable candidate for real-world cryptographic applications involving long-form textual data.

TABLE 6: Long ciphertext with GA-based decryption and mutations 201–300 characters

Table 7 extends the evaluation to an additional long ciphertext, reinforcing the established trends observed in previous analyses. The decrypted message matches the original text, highlighting the precision and effectiveness of the GA-based decryption mechanism. The mutation process continues to demonstrate its role in enhancing security by introducing structured variations in the encrypted output. These mutations contribute to the overall unpredictability of the ciphertext, making it increasingly resistant to cryptographic attacks. The results further support the applicability of the proposed method in diverse encryption scenarios where security and accuracy are paramount. The method’s ability to adapt to different message lengths while preserving encryption integrity ensures its practical utility in securing sensitive communications.

TABLE 7: Long ciphertext with GA-based decryption and mutations 201–300 characters

Table 8 examines the performance of the proposed approach on very long ciphertexts exceeding 301 characters. The results affirm the scalability and reliability of the encryption-decryption framework in handling extensive textual data. The decrypted output accurately reconstructs the original message, demonstrating the robustness of the GA-based key generation and recovery process. The mutations applied to the ciphertext introduce calculated variations, reinforcing encryption security. These systematic alterations ensure that even minor differences in the key result in significantly distinct ciphertext outputs, a property essential for preventing decryption by unauthorized entities. The findings confirm the proposed method is well-suited for high-security applications, where message integrity and confidentiality are critical. By successfully encrypting, decrypting, and mutating long messages, the technique establishes itself as a versatile solution for modern cryptographic challenges.

TABLE 8: Very long ciphertext with GA-based decryption and mutations of 301+characters

Table 9 presents a comparative analysis of the encryption and decryption times for different instances using a genetic optimization-based algorithm and a particle swarm optimization-based algorithm. The results indicate that the particle swarm optimization-based algorithm consistently exhibits slightly higher encryption and decryption times across all instances. For instance, in the first case, the encryption time for the genetic optimization-based approach is 0.000108957 s, whereas for the particle swarm optimization-based approach, it increases to 0.000158957 s. Similarly, the decryption time follows the same trend, with 0.000183343 s for the genetic optimization-based algorithm and 0.000253343 s for the particle swarm-based approach. This pattern persists across all instances, with increasing encryption and decryption times as the instance number grows. The key generation time is also reported, with the genetic optimization-based algorithm achieving 0.453506 s, whereas the particle swarm optimization-based algorithm requires 0.553506 s, highlighting a notable computational overhead. These findings suggest that while both optimization techniques enable secure encryption and decryption, the genetic optimization-based method demonstrates slightly superior efficiency in terms of execution time. This difference in computational performance may have implications for real-time security applications, where minimizing encryption and decryption times is crucial for maintaining system responsiveness.

TABLE 9: Performance evaluation of encryption and decryption times for different instances

In conclusion, the results presented in the tables and figures provide a comprehensive evaluation of the proposed method’s performance in encryption, decryption, and mutation processes. The method demonstrates high accuracy in decryption, robust security enhancements through ciphertext mutations, and scalability across varying message lengths. The effectiveness of the GA in optimizing encryption keys ensures a high degree of security while maintaining computational efficiency. These findings establish the proposed approach as a reliable and practical cryptographic solution suitable for diverse applications requiring secure communication and data protection.

5. CONCLUSION AND FUTURE WORKS

The rapid advancement of quantum computing necessitates the development of robust post-quantum cryptographic solutions to replace traditional cryptographic methods that are vulnerable to quantum attacks. LBC has emerged as a leading candidate due to its resistance to quantum algorithms such as Shor’s. However, the practical implementation of LBC is hindered by challenges related to key size, memory requirements, and computational complexity. In this study, we introduced a GA-based optimization framework that enhances the key generation process in lattice-based encryption. The results demonstrate that the proposed GA-driven approach optimizes key structures, reducing computational overhead while maintaining high-security standards. By refining the selection, crossover, and mutation mechanisms, the GA ensures the generation of highly secure and computationally efficient encryption keys.

The findings of this study indicate that integrating GAs into PQC methodologies can significantly improve performance and resource efficiency. The optimized key generation process enhances encryption speed. It reduces memory footprint, making LBC more viable for real-world applications, particularly in IoT and embedded systems with limited computational resources. Moreover, the GA-based approach introduces an additional layer of randomness and robustness, ensuring that cryptographic keys maintain high entropy and security resilience. However, GAs have inherent limitations. As a heuristic approach, GAs do not guarantee global optimality and rely on extensive parameter tuning (e.g., population size, and mutation rates) to achieve desirable results.

For future research, further refinements can be made by exploring hybrid optimization techniques that integrate GAs with other metaheuristic approaches, such as particle swarm optimization or simulated annealing. In addition, implementing deep learning-driven GAs may further enhance the adaptability and performance of the proposed framework. Expanding the scope of experimental evaluations to real-world cryptographic applications, including secure communication protocols and blockchain security, will provide deeper insights into the practicality and scalability of the method. Furthermore, investigating hardware acceleration techniques, such as FPGA or GPU-based implementations, may improve computational efficiency for large-scale cryptographic operations. The proposed GA-optimized LBC framework can be further refined by addressing these avenues to meet the evolving demands of secure communication in the post-quantum era.