Summary: Our focus lies in developing and analyzing reliable optimization and machine learning algorithms that cater to the needs of contemporary applications. Our primary goal is to develop computationally efficient tools and algorithms that prioritize robustness, fairness, and privacy. By considering these critical factors, our group aims to ensure that our optimization algorithms meet the requirements of trustworthy and responsible data-driven systems. More recently, our group focused on the development of scalable algorithms for generative AI to democratize the training procedure of such large models.
Funding Support: National Science Foundation, National Institutes of Health, Air Force Office of Scientific Research, California Department of Transportation, National Center for Sustainable Transportation, Google, Meta, Amazon, 3M, USC Provost Office, USC Center for Sustainability Solutions
Recent Projects:
Differentially Private Optimization and Learning: Computing solutions to optimization problems are vital for answering science and engineering questions. These problems help find the best design parameters, leading to system or model development. However, adversaries can gain insights into private data by observing system/model parameters. For example, a trained machine learning model’s behavior reveals private training data, and smart grid electricity prices expose power consumption patterns. Developing efficient (differentially) private optimization algorithms is crucial to protect data, ensuring adversaries can’t uncover it by observing algorithm output. In our research, we delve into the fundamental limits of differentially private optimization/learning and focus on the development of efficient algorithms for differentially private learning in both centralized and federated learning settings:
A. Lowy, Z. Li, T. Huang, and M. Razaviyayn “Optimal Differentially Private Learning with Public Data,” available on arXiv.
A. Lowy and M. Razaviyayn, “Private Federated Learning Without a Trusted Server: Optimal Algorithms for Convex Losses,” ICLR 2023.
A. Lowy, A. Ghafelebashi, and M. Razaviyayn, “Private Non-Convex Federated Learning Without a Trusted Server,” AISTATS 2023.
A. Lowy, D. Gupta, and M. Razaviyayn, “Stochastic Differentially Private and Fair Learning,” ICLR 2023.
A.Lowy and M. Razaviyayn, “Private Stochastic Optimization With Large Worst-Case Lipschitz Parameter: Optimal Rates for (Non-Smooth) Convex Losses and Extension to Non-Convex Losses,” ALT 2023.
Min-Max Optimization and Robust Learning: There has been a growing interest in addressing min-max optimization problems within the field of machine learning, particularly in robust learning scenarios where adversarial attacks or distribution shifts are a concern. Although extensive research has been conducted on convex-concave min-max problems, our understanding of the nonconvex regime remains limited. In a series of publications, we delve into the problem of solving non-convex min-max optimization problems and explore the iteration complexity necessary to find specific stationary solutions. Additionally, we apply and analyze these findings in the context of robust learning to develop machine learning algorithms that offer convergence and statistical guarantees. Here are a few sample publications from our research:
S. Baharlouei, F. Sheikholeslami, M. Razaviyayn, Z. Kolter, “Improving Adversarial Robustness via Joint Classification and Multiple Explicit Detection Classes,” AISTATS 2023.
T. Huang, S. Halbe, C. Sankar, P. Amini, S. Kottur, A. Geramifard, M. Razaviyayn, A. Beirami, “Robustness through Data Augmentation Loss Consistency,” TMLR 2023.
S. Baharlouei, K. Ogudu, S.C. Suen, and M. Razaviyayn, “RIFLE: Robust Inference from Low Order Marginals,” TMLR 2023
D. Ostrovskii, B. Barazandeh, M. Razaviyayn, “Nonconvex-nonconcave min-max optimization with a small maximization domain, arXiv 2023.
Z. Wang, K. Balasubramanian, S. Ma, and M. Razaviyayn, “Zeroth-Order Algorithms for Nonconvex Minimax Problems with Improved Complexities,” Journal of Global Optimization 2022.
D. Ostrovskii, A. Lowy, and M. Razaviyayn, “Efficient search of first-order Nash equilibria in nonconvex-concave smooth min-max problems,” SIAM Journal on Optimization, 2021.
M. Razaviyayn, S. Lu, M. Nouiehed, T. Huang, M. Sanjabi, and M. Hong, “Non-convex Min-Max Optimization: Applications, Challenges, and Recent Theoretical Advances,” IEEE Signal Processing Magazine, 2020.
B. Barazandeh and M. Razaviyayn, “Solving Non-convex Non-differentiable Min-Max Games Using Proximal Gradient Method,” ICASSP 2020.
M. Nouiehed, M. Sanjabi, T. Huang, J. D. Lee, and M. Razaviyayn “Solving a Class of Non-Convex Min-Max Games UsingIterative First Order Methods,” NeurIPS 2019.
B. Barazandeh, M. Razaviyayn, and M. Sanjabi, “Training generative networks using random discriminators,” Best Paper Award IEEE Data Science Workshop 2019.
M. Sanjabi, J. Ba, M. Razaviyayn, and J. D. Lee. “On the Convergence and Robustness of Training GANs with Regularized Optimal Transport,” NeurIPS 2018,
Scalable Fair Learning: As machine learning algorithms are deployed in vital decision-making systems impacting human lives, there is a growing concern about potential discrimination based on sensitive attributes like gender or race. Minimizing training/test error alone can lead to systematic biases. Therefore, it is essential to comprehend biases and create fair machine-learning algorithms. Through collaboration with industry, we have developed efficient algorithms that ensure statistical fairness using established fairness metrics while also providing convergence guarantees. Here are a few sample publications from our work:
S. Baharlouei and M. Razaviyayn, “Dr. FERMI: A Stochastic Distributionally Robust Fair Empirical Risk Minimization Framework,” available on arXiv 2023.
A. Lowy, D. Gupta, and M. Razaviyayn, “Stochastic Differentially Private and Fair Learning,” ICLR 2023.
A. Lowy*, S. Baharlouei*, R. Pavan, M. Razaviyayn, and A. Beirami. “A stochastic optimization framework for fair risk minimization,” TMLR 2023.
S. Baharlouei, M. Nouiehed, and M. Razaviyayn, “Rényi Fair Inference,” ICLR 2020.
Federated Learning and Distributed Optimization: Federated learning (FL) is a collaborative learning approach where multiple clients/users with diverse data collaborate to train a model. The distributed optimization framework plays a vital role in federated learning. Our research focuses on developing efficient distributed algorithms, including privacy-preserving ones, for FL. We investigate the fundamental limits of computing different notions of stationarity under privacy constraints, considering privacy notions like central differential privacy, local differential privacy, and shuffled differential privacy. Here are a few sample publications from our work:
A. Lowy and M. Razaviyayn, “Private Federated Learning Without a Trusted Server: Optimal Algorithms for Convex Losses,” ICLR 2023.
A. Lowy, A. Ghafelebashi, and M. Razaviyayn, “Private Non-Convex Federated Learning Without a Trusted Server,” AISTATS 2023.
S. Lu, J. Lee, M. Razaviyayn, M. Hong, “Linearized ADMM Converges to Second-Order Stationary Points for Non-Convex Problems,” IEEE TSP 2021.
M. Hong, M. Razaviyayn, and J. D. Lee, “Gradient primal-dual algorithm converges to second-order stationary solution for nonconvex distributed optimization over networks,” ICML 2018.
M. Hong, Z.-Q. Luo, and M. Razaviyayn. “Convergence analysis of alternating direction method of multipliers for a family of nonconvex problems,” SIAM Journal on Optimization 2016.
M. Razaviyayn, M. Hong, Z.-Q. Luo, JS. Pang, “Parallel successive convex approximation for nonsmooth nonconvex optimization,” NeurIPS 2014.
We also studied some applications of FL in traffic congestion reduction:
A. Ghafelebashi, M. Razaviyayn, M. Dessouky, “Congestion reduction via personalized incentives,” Transportation Research Part C: Emerging Technologies, 2023.
A. Ghafelebashi, M. Razaviyayn, M. Dessouky, “Incentive Systems for New Mobility Services,” METRANS Transportation Center Report 2022.
Toward Finding Global Optima in Certain Nonconvex Optimization Problems: To train advanced machine learning models like deep neural networks or matrix completion, nonconvex optimization problems must be solved. Understanding the landscape of these optimization problems is crucial. Additionally, the optimization algorithm should incorporate modules that can navigate away from misleading saddle points and discover higher-order stationary solutions. Our research involves analyzing the landscape of popular nonconvex optimization problems and devising algorithms that effectively escape saddle points and find higher-order stationary solutions. Through a series of studies, we have made notable contributions in this area:
Y. Han, M. Razaviyayn, R. Xu, “Policy Gradient Converges to the Globally Optimal Policy for Nearly Linear-Quadratic Regulators,” arXiv 2022.
M. Nouiehed and M. Razaviyayn. “Learning deep models: Critical points and local openness,” INFORMS Journal on Optimization 2022.
S. Lu, J. D. Lee, M. Razaviyayn, and M. Hong, “Linearized ADMM Converges to Second-Order Stationary Points for Non-Convex Problems,” IEEE TSP 2021.
M. Nouiehed and M. Razaviyayn, “A Trust Region Method for Finding Second-Order Stationarity in Linearly Constrained Nonconvex Optimization,” SIAM Journal on Optimization 2020.
S. Lu, M. Razaviyayn, B. Yang, K. Huang, M. Hong, “Finding second-order stationary points efficiently in smooth nonconvex linearly constrained optimization problems,” Spotlight Presentation at NeurIPS 2020.
M. Hong, M. Razaviyayn, and J. Lee. “Gradient primal-dual algorithm converges to second-order stationary solution for nonconvex distributed optimization over networks,” ICML 2018.
Variance Amplification of First-Order Methods: First-order algorithms are popular in machine learning and for large-scale optimization problems due to their simplicity and scalability. However, when the gradient is not precisely available, such as in cases involving costly simulations or noisy measurements, first-order algorithms rely on noisy gradient estimates. In our research, we analyze the performance of first-order algorithms when the gradient is perturbed by additive white noise. In collaboration with Hesameddin Mohammadi and Mihailo Jovanovic, we establish upper bounds on the noise amplification of various popular first-orde algorithms. We also established fundamental lower bounds on the performance of certain types of first-order methods. Additionally, we study the influence of network topology and spatial dimension on the performance of first-order algorithms in distributed computation scenarios:
H. Mohammadi, M. Razaviyayn, and M. R. Jovanović, “Tradeoffs between convergence rate and noise amplification for momentum-based accelerated optimization algorithms,” arXiv 2022.
H. Mohammadi, M. Razaviyayn, and M. R. Jovanović, “Robustness of accelerated first-order algorithms for strongly convex optimization problems,” IEEE Transactions on Automatic Control 2020.
H. Mohammadi, M. Razaviyayn, and M. R. Jovanović, “Performance of noisy Nesterov’s accelerated method for strongly convex optimization problems,” ACC 2019.
H. Mohammadi, M. Razaviyayn, and M. R. Jovanović, “Variance amplification of accelerated first-order algorithms for strongly convex quadratic optimization problems,” CDC 2018.
Axiomatic Approach to Machine Learning Explainability: The growing effectiveness of deep learning (DL) has raised concerns about the lack of model explainability. Determining suitable methods for explaining these models remains uncertain. To address this, we have conducted a series of studies focused on identifying approaches that fulfill specific axioms for machine learning explainability. By finding methods that uniquely satisfy these desired axioms, we aim to contribute to the development of reliable and trustworthy explanations for DL models.
D. Lundstrom and M. Razaviyayn, “Four Axiomatic Characterizations of the Integrated Gradients Attribution Method,” available on arXiv 2023.
D. Lundstrom and M. Razaviyayn, “Distributing Synergy Functions: Unifying Game-Theoretic Interaction Methods for Machine-Learning Explainability,” ICML 2023.
D. Lundstrom, T. Huang, and M. Razaviyayn, “A Rigorous Study of Integrated Gradients Method and Extensions to Internal Neuron Attributions,” ICML 2022.
