A Novel Class of Hybrid Conjugate Gradient Methods for Unconstrained Optimization with Applications to Image Denoising
DOI:
https://doi.org/10.21928/uhdjst.v9n2y2025.pp166-183Keywords:
Conjugate Gradient Method, Strong Wolfe Condition, Global Convergence, Unconstrained Optimization, Convex Combination, Hybrid Method, Image ProcessingAbstract
This paper presents a new hybrid conjugate gradient (CG) method for solving large-scale unconstrained optimization problems where classical CG algorithms may not perform well. This new method integrates four classical algorithms, namely, Liu and Storey, Fletcher and Reeves, Dai and Yuan, and Polak, Ribiere, and Polyak, using a convex combination of CGs and an inexact line search based on strong Wolfe conditions, and adheres to the Dai-Liao conjugate condition to enhance convergence properties. The theoretical study established the conditions for sufficient descent and global convergence of the hybrid CG method. Numerical studies demonstrated that the proposed method significantly reduced the number of iterations and computational time, achieving superior evaluation efficiency compared with other CG methods. In particular, the effectiveness of the proposed algorithm is validated for image impulse denoising, where it can recover high-quality images while preserving significant features, implying its practical applicability to real-world signal and image processing problems.
References
W. W. Hager and H. Zhang. “A survey of nonlinear conjugate gradient methods”. Pacific Journal of Optimization, vol. 2, no. 1, pp. 35-58, 2006.
M. R. Hestenes and E. Stiefel. “Methods of Conjugate Gradients for Solving Linear Systems”. vol. 49. NBS, Washington, DC, 1952.
H. A. Babando, M. Barma, S. Nasiru and S. Suraj. “A dai-liao hybrid prp and dy schemes for unconstrained optimization”. International Journal of Development Mathematics, vol. 1, no. 1, pp. 41-53, 2024.
P. Wolfe. “Convergence conditions for ascent methods”. SIAM Review, vol. 11, no. 2, pp. 226-235, 1969.
P. Wolfe. “Convergence conditions for ascent methods. II: Some corrections”. SIAM Review, vol. 13, no. 2, pp. 185-188, 1971.
R. Fletcher and C. M. Reeves. “Function minimization by conjugate gradients”. The Computer Journal, vol. 7, no. 2, pp. 149-154, 1964.
E. Polak and G. Ribiere. “Note sur la convergence de méthodes de directions conjuguées. Revue française d’informatique et de recherche opérationnelle”. Série Rouge, vol. 3, no. 16, pp. 35-43, 1969.
B. T. Polyak. “The conjugate gradient method in extremal problems”. USSR Computational Mathematics and Mathematical Physics, vol. 9, no. 4, pp. 94-112, 1969.
R. Fletcher. “Practical Methods of Optimization”. John Wiley and Sons, Hoboken, 2000.
Y. Liu and C. Storey. “Efficient Generalized conjugate gradient algorithms, part 1: Theory”. Journal of Optimization Theory and Applications, vol. 69, pp. 129-137, 1991.
Y. H. Dai and Y. Yuan. “A nonlinear conjugate gradient method with a strong global convergence property”. SIAM Journal on Optimization, vol. 10, no. 1, pp. 177-182, 1999.
J. Nocedal and S. J. Wright. “Numerical Optimization”. Springer, Berlin, 2006.
N. Andrei. “Numerical comparison of conjugate gradient algorithms for unconstrained optimization”. Studies in Informatics and Control, vol. 16, no. 4, pp. 333-352, 2007.
M. J. D. Powell. Nonconvex Minimization Calculations and the Conjugate Gradient Method. In: “Numerical Analysis: Proceedings of the 10th Biennial Conference Held at Dundee, Scotland, June 28-July 1, 1983”, Springer, pp. 122-141, 1984.
Y. Dai, J. Han, G. Liu, D. Sun, H. Yin and Y. X. Yuan. “Convergence properties of nonlinear conjugate gradient methods”. SIAM Journal on Optimization, vol. 10, no. 2, pp. 345-358, 2000.
S. Babaie-Kafaki. “A hybrid conjugate gradient method based on a quadratic relaxation of the dai-yuan hybrid conjugate gradient parameter”. Optimization, vol. 62, no. 7, pp. 929-941, 2013.
N. Andrei. “Another nonlinear conjugate gradient algorithm for unconstrained optimization”. Optimization Methods and Software, vol. 24, no. 1, pp. 89-104, 2009.
S. S. Djordjević. “New hybrid conjugate gradient method as a convex combination of FR and PRP methods”. Filomat, vol. 30, no. 11, pp. 3083-3100, 2016.
S. S. Djordjević. “New hybrid conjugate gradient method as a convex combination of LS and FR methods”. Acta Mathematica Scientia, vol. 39. pp. 214-228, 2019.
J. K. Liu and S. J. Li. “New hybrid conjugate gradient method for unconstrained optimization”. Applied Mathematics and Computation, vol. 245, pp. 36-43, 2014.
S. B. Hanachi, B. Sellami and M. Belloufi. “New iterative conjugate gradient method for nonlinear unconstrained optimization”. RAIRO-Operations Research, vol. 56, no. 4, pp. 2315-2327, 2022.
S. B. Hanachi, B. Sellami and M. Belloufi. “A new family of hybrid conjugate gradient method for unconstrained optimization and its application to regression analysis”. RAIRO-Operations Research, vol. 58, no. 1. pp. 613-627, 2024.
N. Salihu, M. R. Odekunle, A. M. Saleh, S. Salihu. “A dai-liao hybrid hestenes-stiefel and fletcher-revees methods for unconstrained optimization”. International Journal of Industrial Optimization, vol. 2, no. 1, pp. 33-50, 2021.
A. Hallal, M. Belloufi and B. Sellami. “An efficient new hybrid cg-method as convex combination of DY and CD and HS algorithms”. RAIRO-Operations Research, vol. 56, no. 6, pp. 4047-4056, 2022.
A. Perry. “A modified conjugate gradient algorithm”. Operations Research, vol. 26, no. 6, pp. 1073-1078, 1978.
Y. H. Dai and L. Z. Liao. “New conjugacy conditions and related nonlinear conjugate gradient methods”. Applied Mathematics and Optimization, vol. 43, pp. 87-101, 2001.
A. B. Abubakar and P. Kumam. “A descent dai-liao conjugate gradient method for nonlinear equations”. Numerical Algorithms, vol. 81, pp. 197-210, 2019.
N. Andrei. “A dai-liao conjugate gradient algorithm with clustering of eigenvalues”. Numerical Algorithms, vol. 77, no. 4, pp. 1273-1282, 2018.
S. Babaie-Kafaki. “On optimality of two adaptive choices for the parameter of dai-liao method”. Optimization Letters, vol. 10, pp. 1789-1797, 2016.
S. Babaie-Kafaki and R. Ghanbari. “The dai-liao nonlinear conjugate gradient method with optimal parameter choices”. European Journal of Operational Research, vol. 234, no. 3, pp. 625-630, 2014.
S. Babaie-Kafaki and R. Ghanbari. “A descent family of dai-liao conjugate gradient methods”. Optimization Methods and Software, vol. 29, no. 3, pp. 583-591, 2014.
S. Babaie-Kafaki and R. Ghanbari. “Two optimal dai-liao conjugate gradient methods”. Optimization, vol. 64, no. 11, pp. 2277-2287, 2015.
S. Babaie-Kafaki and R. Ghanbari. “Two adaptive dai-liao nonlinear conjugate gradient methods”. Iranian Journal of Science and Technology, Transactions A: Science, 42, pp. 1505-1509, 2018.
B. Ivanov, G. V. Milovanović and P. S. Stanimirović. “Accelerated dai-liao projection method for solving systems of monotone nonlinear equations with application to image deblurring”. Journal of Global Optimization, vol. 85, no. 2, pp. 377-420, 2023.
M. Y. Waziri, K. Ahmed and J. Sabi’u. “A dai-liao conjugate gradient method via modified secant equation for system of nonlinear equations”. Arabian Journal of Mathematics, vol. 9, pp. 443-457, 2020.
A. B. Abubakar, P. Kumam, M. Malik and A. H. Ibrahim. “A hybrid conjugate gradient based approach for solving unconstrained optimization and motion control problems”. Mathematics and Computers in Simulation, vol. 201, pp. 640-657, 2022.
A. B. Abubakar, P. Kumam, M. Malik, P. Chaipunya and A. H. Ibrahim. “A hybrid FR-DY conjugate gradient algorithm for unconstrained optimization with application in portfolio selection”. AIMS Mathematics, vol. 6, no. 6, pp. 6506-6527, 2021.
G. Zoutendijk. “Nonlinear programming, computational methods”. Integer and Nonlinear Programming”, pp. 37-86, 1970.
N. Andrei. “An unconstrained optimization test functions collection”. Advanced Modeling and Optimization, vol. 10, no. 1, pp. 147-161, 2008.
N. I. M, Gould, D. Orban and P. L. Toint. “Cuter and sifdec: A constrained and unconstrained testing environment, revisited”. ACM Transactions on Mathematical Software, vol. 29, no. 4, pp. 373-394, 2003.
J. J. Moré, B. S. Garbow and K. E. Hillstrom. “Testing unconstrained optimization software”. ACM Transactions on Mathematical Software, vol. 7, no. 1, pp. 17-41, 1981.
E. D. Dolan and J. J. Moré. “Benchmarking optimization software with performance profiles”. Mathematical Programming, vol. 91, pp. 201-213, 2002.
R. H. Chan, C. W. Ho and M. Nikolova. “Salt-and-pepper noise removal by median-type noise detectors and detail-preserving regularization”. IEEE Transactions on Image Processing, vol. 14, no. 10, pp. 1479-1485, 2005.
J. F. Cai, R. Chan and B. Morini. Minimization of an Edge-Preserving Regularization Functional by Conjugate Gradient Type Methods. In: “Image Processing Based on Partial Differential Equations: Proceedings of the International Conference on PDEBased Image Processing and Related Inverse Problems”, CMA, Oslo, Springer, pp. 109-122, 2007.
H. Hwang and R. A. Haddad. “Adaptive median filters: New algorithms and results”. IEEE Transactions on Image Processing, vol. 4, no. 4, pp. 499-502, 1995.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2025 Rega Jalal, Hawraz N. Jabbar
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
