Joint PCA

Creator
Creator
Seonglae ChoSeonglae Cho
Created
Created
2025 Nov 13 21:49
Editor
Edited
Edited
2026 Jan 10 16:52
Refs
Refs
 
 
 
 

AI Counting (linebreaking)

notion image
In fixed-width text, the model must predict \n by determining whether the next word exceeds the line boundary. The model represents character count (current line length), total line width, remaining character count, next word length, etc. on a 1-dimensional "feature
Manifold
". This manifold takes a
Helix
shape embedded in a low-rank subspace (≈6D) of the high-dimensional residual stream, with features tiling the manifold. QK rotation (attention) performs boundary detection by rotating and aligning one manifold to another (line width representation). Multiple boundary heads cooperate with different offsets to estimate remaining character count at high resolution. Next word length and remaining character count are represented in nearly orthogonal subspaces, making "line break decision" linearly separable.
Gibbs phenomenon
(ringing) creates a rippled manifold—an oscillatory residual pattern (overshoot–undershoot oscillation such as
Moire
and
Aliasing
) that emerges when projecting signals from high to low dimensions or approximating continuous values in finite dimensions.
 

Recommendations