Unsupervised Machine Learning & Generative AI

Module 1: Core Principles and Parameter Estimation

Unsupervised machine learning methods process data where the output is not explicitly given, with the objective of evaluating the cumulative output on all samples. These models are broadly categorized into two frameworks:

Descriptive Models (Recoding Framework): These models offer an additional description or new representation of the data, such as finding structure, orthogonal variation, or statistical independence. Methods include Principal Component Analysis (PCA) and Independent Component Analysis (ICA).
Generative Models: These aim to simulate the world by approximating the distribution of the data generation process. The goal is to match the observed data density and generate new data that resembles the observed instances. Examples include Factor Analysis, Hidden Markov Models, and restricted Boltzmann machines.

Quality of Parameter Estimation

Generative models aim to estimate true parameters given a parametrized model class.

Bias: The expected difference between the estimated and true parameter.
Mean Squared Error (MSE): The expected squared error between the estimated parameter $\hat{w}$ and the true parameter $w$ , representing the theoretical objective to minimize:

$\text{MSE} = \text{Bias}^2 + \text{Variance}$

Maximum Likelihood Estimator (MLE)

The likelihood $L(w)$ of a dataset $X = \{x_1, ..., x_N\}$ is the probability that the model produces the observed data. For identically and independently distributed (i.i.d) data, the negative log-likelihood is minimized to find optimal parameters:

$-\ln L(w) = -\sum_{i=1}^N \ln p(x_i; w)$

The MLE is asymptotically unbiased, efficient, and consistent, meaning for large training sets, the estimator approaches the true parameter value with decreasing variance.

Maximum Entropy

The maximum entropy principle looks for the probability distribution that assumes minimal prior knowledge while satisfying specific constraints, meaning it is the most likely observed solution. The solution is given by the Gibbs distribution:

$p(x) = \frac{1}{Z} \exp\left( -\sum_i \lambda_i f_i(x) \right)$

where $Z$ is the partition function and $\lambda_i$ are the Lagrange multipliers determined by the constraint equation system.

Module 2: The Expectation Maximization (EM) Algorithm

When the likelihood cannot be computed analytically (due to hidden states, non-linearities, or many-to-one output mappings), the Expectation Maximization algorithm is utilized. The likelihood is determined by all unobserved/hidden variables $u$ mapped to observations $x$

Because the joint probability $p(x, u; w)$ is often easier to compute than the marginal likelihood, EM estimates the likelihood using a lower bound driven by Jensen's inequality.

E-step (Expectation): Estimates the posterior distribution of the hidden variables given the current parameters. It does not change the parameters but maximizes the lower bound.
M-step (Maximization): Updates the model parameters to maximize the expected log joint probability (the lower bound) formulated in the E-step.

Module 3: Factor Analysis, PCA, and ICA

These are primary projection methods used to summarize multivariate data or map it to lower-dimensional spaces while maintaining neighbourhoods.

Factor Analysis (FA)

Factor analysis is a generative model modelling observations $x$ using a hidden factor vector $z$ and additive noise $\epsilon$ The model assumes the factors are independent standard Gaussians, and the noise is independent Gaussian:

$x = \Lambda z + \epsilon$

Where $\Lambda$ is the loading matrix, $z \sim N(0, I)$ , and $\epsilon \sim N(0, \Psi)$ with $\Psi$ being diagonal.
Factor estimation requires the matrix inversion lemma to compute projections:

$E[z|x] = \Lambda^T (\Lambda \Lambda^T + \Psi)^{-1} x$

EM updates are used to optimize $\Lambda$ and $\Psi$ recursively. FA differs from PCA because it assumes additive noise and isn't scale-invariant, whereas PCA assumes no noise and ranks strictly by eigenvalues.

Principal Component Analysis (PCA) vs Independent Component Analysis (ICA)

PCA: Projects to a lower-dimensional space under maximal information conservation (variance maximization). It computes orthogonal decorrelated abstractions.
ICA: Projects into a space with statistically independent components. It optimizes characteristics of factorial distributions by maximizing entropy for a given variance or matching super-Gaussian prototypes. ICA is unique up to scale and permutation, whereas PCA provides a rigidly ranked eigenvalue structure.

Module 4: Scaling, Projection & Manifold Learning

Multidimensional Scaling (MDS)

MDS projects data to a low-dimensional space while preserving distances between points. The objective is to match the embedded distance $d_{ij}$ to the original distance $\delta_{ij}$

Sammon Mapping: A non-linear mapping that heavily penalizes fractional/relative errors for small distances.
Kruskal's Measure: A non-metric MDS penalizing large errors.

Non-negative Matrix Factorization (NMF)

NMF is a matrix factorization method enforcing non-negativity constraints, making it a purely additive (parts-based) representation. NMF relies on two primary objectives:

Kullback-Leibler Divergence: Minimizes $D(V || \Lambda Z)$ using gradient descent, resulting in multiplicative update rules.
Euclidean Distance (Frobenius Norm): Minimizes $\|X - \Lambda Z\|_F^2$

Locally Linear Embedding (LLE) & Isomap

LLE: Preserves local neighborhood topology by reconstructing each point from its neighbors $X_j$ using linear weights $W_{ij}$ The optimal embedding coordinates $Y$ correspond to the bottom $d$ eigenvectors of the matrix $(I - W)^T (I - W)$
Isomap: Preserves global geometry by preserving geodesic distances. The coordinates are calculated as the top $l$ eigenvectors of the doubly centered geodesic distance matrix $\tau(D) = -\frac{1}{2} H S H$ , where $S$ contains squared geodesic distances and $H$ is the centering matrix.

t-SNE & Self-Organizing Maps (SOM)

t-SNE: A stochastic neighbor embedding algorithm that combats the "crowding" problem when embedding highly complex data by utilizing a heavy-tailed t-distribution.
SOM: Equidistantly fills a hypercube representing the data space. The online update rule relies on a learning rate $\eta$ and a window function $h(i, a^*)$ that decreases with distance to the active unit:

$w_i(t+1) = w_i(t) + \eta(t) h(i, a^*, t) [x(t) - w_i(t)]$

Module 5: Clustering & Biclustering

Mixture of Gaussians (MoG)

Mixtures model data generatively, where $\pi_j$ is the probability of choosing component $j$ The Maximum A Posteriori (MAP) approach applies priors like Dirichlet (weights) and Wishart (covariances). The EM updates involve computing the posterior probability $\gamma_{ij}$ that point $i$ belongs to component $j$ , then updating the parameters $\mu_j, \Sigma_j, \pi_j$ Constraining covariance matrices can specify clusters as spherical, diagonal, or ellipsoidal with equal or varying volumes and shapes.

k-Means & Affinity Propagation

Fuzzy k-Means: Uses continuous softmax memberships.
Affinity Propagation: Uses similarity matrices $s(i, j)$ (e.g., negative Euclidean distance) to send messages iteratively between points until exemplars emerge, which is particularly effective for spherical clusters.

FABIA (Biclustering)

Factor Analysis for Bicluster Acquisition (FABIA) models $l$ biclusters and additive noise generatively. FABIA uses sparse prototype and factor matrices, exploiting the formula $X = \Lambda Z + \Upsilon$ with independent Gaussian noise $\Upsilon$ By encouraging sparse solutions, genes and conditions strongly associated with a specific biological block are isolated.

Module 6: Sequential and Energy-Based Models

Hidden Markov Models (HMM)

HMMs are highly suited for discrete sequential data like DNA and proteins. The joint probability depends on the first-order Markov assumption.

Forward Algorithm: Computes the probability of sequence $X$ recursively over time $T$
Viterbi Algorithm: Recursively finds the most likely hidden state sequence via backtracing.
Baum-Welch (EM): Derives estimation variables $\alpha$ (forward) and $\beta$ (backward suffix sequences) to compute maximum likelihood parameters iteratively.

Boltzmann Machines & RBMs

Energy-based models where the global states with the highest probabilities get the lowest energies. The energy difference relies on a temperature parameter $T$ , effectively incorporating the Boltzmann constant.

Restricted Boltzmann Machine (RBM): Restricts connectivity ensuring no intra-layer connections.
Learning (Contrastive Divergence): Approximates the Kullback-Leibler objective using local information. The weight update is calculated via the positive gradient (data distribution) minus the negative gradient (reconstructed distribution):

$\Delta W_{ij} \propto \langle v_i h_j \rangle_{\text{data}} - \langle v_i h_j \rangle_{\text{model}}$