Seonglae ChoModel Sparsity Overview
Model sparsity reduces Model Complexity by eliminating parameters that are non-zero but meaningless, effectively decreasing the number of parameters that contribute little to model performance.
Regularization Techniques
- L0 Norm directly counts non-zero parameters and induces sparsity by encouraging parameters to become exactly zero in the model's loss function.
- L1 Norm promotes sparsity by penalizing the absolute values of parameters. However, it is not differentiable at zero, which can pose challenges for gradient-based optimization methods.
Model Sparsity Notion
