Seonglae ChoAssigns Bayesian weights to all possible programs to predict future data without iid assumption. While theoretically the most powerful prediction method, it's uncomputable and requires approximation.
Solomonoff Induction considers all hypotheses as candidates, assigning higher weights to simpler hypotheses based on their Kolmogorov Complexity, providing a mathematical justification for Occam's Razor. For observed data, it evaluates the probability of all possible programs that could generate that data, with simpler programs being considered more likely. This implements an ideal Bayesian approach to predicting future data.
While theoretically powerful, it is computationally impossible to implement in practice since it requires considering all possible programs. Instead, this model serves as a theoretical limit and ideal standard for inference.
An Intuitive Explanation of Solomonoff Induction — LessWrong
This is the completed article that Luke wrote the first half of. My thanks go to the following for reading, editing, and commenting; Luke Muehlhauser…
https://www.lesswrong.com/posts/Kyc5dFDzBg4WccrbK/an-intuitive-explanation-of-solomonoff-induction
Solomonoff induction - LessWrong
Solomonoff induction is an ideal answer to questions like "What probably comes next in the sequence 1, 1, 2, 3, 5, 8?" or "Given the last three years of visual data from this webcam, what will this robot probably see next?" or "Will the sun rise tomorrow?" Solomonoff induction requires infinite computing power, and is defined by taking every computable algorithm for giving a probability distribution over future data given past data, weighted by their algorithmic simplicity, and updating those weights by comparison to the actual data. E.g., somewhere in the ideal Solomonoff distribution is an exact copy of you, right now, staring at a string of 1s and 0s and trying to predict what comes next - though this copy of you starts out with a very low weight in the mixture owing to its complexity. Since a copy of you is present in this mixture of computable predictors, we can prove a theorem about how well Solomonoff induction does compared to an exact copy of you; namely, Solomonoff induction commits only a bounded amount of error relative to you, or any other computable way of making predictions. Solomonoff induction is thus a kind of perfect or rational ideal for probabilistically predicting sequences, although it cannot be implemented in reality due to requiring infinite computing power. Still, considering Solomonoff induction can give us important insights into how non-ideal reasoning should operate in the real world. Additional reading: * https://en.wikipedia.org/wiki/Solomonoff%27s_theory_of_inductive_inference * http://lesswrong.com/lw/dhg/an_intuitive_explanation_of_solomonoff_induction/ * http://wiki.lesswrong.com/wiki/Solomonoff_induction
https://www.lesswrong.com/w/solomonoff-induction
Solomonoff's theory of inductive inference
Solomonoff's theory of inductive inference proves that, under its common sense assumptions (axioms), the best possible scientific model is the shortest algorithm that generates the empirical data under consideration. In addition to the choice of data, other assumptions are that, to avoid the post-hoc fallacy, the programming language must be chosen prior to the data[1] and that the environment being observed is generated by an unknown algorithm. This is also called a theory of induction. Due to its basis in the dynamical (state-space model) character of Algorithmic Information Theory, it encompasses statistical as well as dynamical information criteria for model selection. It was introduced by Ray Solomonoff, based on probability theory and theoretical computer science.[2][3][4] In essence, Solomonoff's induction derives the posterior probability of any computable theory, given a sequence of observed data. This posterior probability is derived from Bayes' rule and some universal prior, that is, a prior that assigns a positive probability to any computable theory.
https://en.wikipedia.org/wiki/Solomonoff%27s_theory_of_inductive_inference
Pre-Training GPT-4.5
Sam Altman sits down with Amin Tootoonchian, Alex Paino, and Daniel Selsam to discuss the journey to get to GPT-4.5.
https://www.youtube.com/watch?v=6nJZopACRuQ&t=39m
