Seonglae ChoRecovering the unknown latent variables and the generative function from observed data is an ill-posed problem without additional assumptions. Prior work has typically assumed linearity or relied on auxiliary supervision, but such assumptions are hard to validate in real-world settings, and theoretical guarantees can break when the assumptions are violated. This work aims to identify the key ingredients that can still be guaranteed even in general settings where full identifiability is not ensured.
This work introduces a new problem, Diverse Dictionary Learning, and proves that if the dependency structure between latent and observed variables is sufficiently diverse, one can identify set-theoretic relationships among latent variables without strong parameter constraints. In particular, the intersection, complement, and symmetric difference of the latent variables connected to an arbitrary set of observations are identifiable up to an appropriate level of indeterminacy. These results offer a principled way to understand the structure of a hidden world, in the spirit of the classical genus–differentia definition.
To realize these identifiability benefits in practice, the method introduces a simple inductive bias that encourages sparsity of the Jacobian during estimation. Unlike Sparse Autoencoders (SAEs), which impose sparsity on the latent variables themselves and can suffer side effects such as feature splitting, this approach focuses on sparsity in the dependency structure. This leads to more interpretable and stable representations. The proposed regularizer can be easily integrated into the model objective as a Jacobian loss:
Empirically, on synthetic datasets, the proposed method achieves a substantially higher Mean Correlation Coefficient (MCC) than prior Jacobian/Hessian-penalty-based methods, with the advantage growing as dimensionality increases. In a 4D setting, OroJAR achieves an MCC of 0.63, whereas this method reaches 0.84, demonstrating that the theory translates into practical gains. On real image datasets such as Cars3D and MPI3D, adding the regularizer to FactorVAE also yields significant improvements in both the FactorVAE score and the DCI metric.
arxiv.org
https://arxiv.org/pdf/2604.17568