Seonglae ChoGradient descent used for constrained optimization
- First, take a gradient descent step.
- If this step leaves the feasible set, project back onto the constraint set.
Here, is the allowed region (the constraint set), and is the projection operator that maps a point back into that region. In other words, you move in the direction that decreases the loss, but if you step outside the allowed region, you return to the closest feasible point.