If you were to invent statistics from scratch, how would I do it? Statistics can be seen as a "converse" of probability, and it is essentially a field that branches out from math. In probability, one takes a distribution and attempts to describe what the samples look like. In statistics, we are given the samples first and then try to infer what the distribution is. This is usually an extremely difficult problem, and so rather than trying to describe the entire distribution, we try to talk about certain parameters about the distribution (e.g. what is the mean, variance?).

At the heart of statistics is the seemingly unrelated information theory, which talks about how much information (e.g. bits) can be transmitted through a noisy channel. The concepts of entropy and KL-divergence provide a good transition between probability and statistics. At this point, we can branch off into two paradigms of inference. First is the frequentist approach, which attempts to model the parameter as a random variable that realizes through a sampling distribution. The second is the Bayesian approach, which assumes a prior distribution on the data, which with the data and Bayes rule gets updated to our posterior.

Once the foundations of these two theories are established, the main applications lie in machine learning, which uses computer science to design algorithmic approaches to statistical inference. This field can be seen as an integration of statistics and computer science, since we heavily use the theory of algorithms to optimize objective functions that are determined by statistics (e.g. greedy algorithms to fit decision trees, L1 regularization as a convex approximation of best subset regression). In fact, optimization is such an important subset of machine learning that it deserves its own set of notes. In here, I go through the main concepts in convex optimization, along with an index of other non-convex methods. It turns out that the fields of optimization and sampling (which is also used for numerical integration) are heavily related, as slight modifications of optimizers lead to samplers (e.g. SGD vs SGLD).

If you were to invent statistics from scratch, how would I do it? Statistics can be seen as a "converse" of probability, and it is essentially a field that branches out from math. In probability, one takes a distribution and attempts to describe what the samples look like. In statistics, we are given the samples first and then try to infer what the distribution is. This is usually an extremely difficult problem, and so rather than trying to describe the entire distribution, we try to talk about certain parameters about the distribution (e.g. what is the mean, variance?).

At the heart of statistics is the seemingly unrelated information theory, which talks about how much information (e.g. bits) can be transmitted through a noisy channel. The concepts of entropy and KL-divergence provide a good transition between probability and statistics. At this point, we can branch off into two paradigms of inference. First is the frequentist approach, which attempts to model the parameter as a random variable that realizes through a sampling distribution. The second is the Bayesian approach, which assumes a prior distribution on the data, which with the data and Bayes rule gets updated to our posterior.

Once the foundations of these two theories are established, the main applications lie in machine learning, which uses computer science to design algorithmic approaches to statistical inference. This field can be seen as an integration of statistics and computer science, since we heavily use the theory of algorithms to optimize objective functions that are determined by statistics (e.g. greedy algorithms to fit decision trees, L1 regularization as a convex approximation of best subset regression). In fact, optimization is such an important subset of machine learning that it deserves its own set of notes. In here, I go through the main concepts in convex optimization, along with an index of other non-convex methods. It turns out that the fields of optimization and sampling (which is also used for numerical integration) are heavily related, as slight modifications of optimizers lead to samplers (e.g. SGD vs SGLD).

With the universal approximation theorem, better engineering, and exponentially-increasing computatonal power, deep neural networks have become extremely powerful models for complex and high-dimensional data. They start out with simple multilayer perceptrons but recent research has pushed the architectures to CNNs, RNNs, LSTMs, energy models, encoder-decoders, flow models, attention layers, and most recently diffusion models. While the models are inherent black-box in nature, several heuristics and architectures have been developed to push their applications to the field of computer vision (CV) and natural language processing (NLP). The most noticeable success of these applications come in autonomous driving and large language models (LLMs).

Finally, another subfield of machine learning is called reinforcement learning, which teaches agents to make decisions through simulations involving trial and error. These models are widely used in robotics and simulations, and this field of optimizing rewards and penalties heavily relies on decision theory.

All of my personal notes are free to download, use, and distrbute under the Creative Commons "Attribution- NonCommercial-ShareAlike 4.0 International" license. Please contact me if you find any errors in my notes or have any further questions.