Hands on AI I

Module 1: Data Foundations and Processing

1.1 Tabular Data and Dimensionality Reduction

Data is often structured in a tabular form, where elements are arranged in vertical columns (features) and horizontal rows (samples). For instance, a dataset containing $n$ samples and $d$ features can be modelled as a matrix $X$, where individual sample vectors are defined as $x = (x_1, \dots, x_d)$ and their corresponding labels as $y$. Due to the high dimensionality of features (e.g., $d=13$ in the Wine dataset), visualizing raw tabular data is often infeasible. Dimensionality reduction techniques like Principal Component Analysis (PCA) and t-Distributed Stochastic Neighbour Embedding (t-SNE) are applied to compress $n$ -dimensional data into lower-dimensional spaces (e.g., $2$ -dimensional) to preserve and visualize clustered variance.

1.2 Image, Sequential, and Categorical Data Representation

Images are multidimensional structures containing height, length (or width), and channels. The RGB model utilizes three colour channels (Red, Green, Blue), whereas the RGBA model includes an alpha ( $\alpha$ ) channel representing transparency. Image data can be processed through augmentation techniques such as rotation, flipping, noise injection, and input dropout to reduce overfitting and increase model robustness.

Sequential data forms like sound waves rely on continuous signals measured in Hertz (Hz). To accurately reconstruct a sound signal computationally, the Nyquist-Shannon theorem must be satisfied, which dictates that the sampling rate $f_s$ must exceed twice the highest signal frequency:

$$f_s > 2f$$

Alternatively, sampling points must be separated by $< \frac{1}{2f}$ seconds. Furthermore, Fourier transformations decompose signals from the time domain into the frequency domain, visualized via spectrograms representing energy over time.

Categorical textual data must map distinct words into mathematical vectors. Simple mapping relies on one-hot encoding, producing a binary vector $v = (v_1, \dots, v_n)$ where $v_i = 1$ if category $i$ is true, and $0$ otherwise. To handle vast vocabularies, Word Embeddings represent words in smaller, continuous vector spaces containing semantic similarities.

Module 2: Unsupervised Learning

Unsupervised learning segments unlabelled datasets ( $X$ ) into structural partitions relying solely on input features.

2.1 Similarity Metrics

Similarity is computed via distance measures. The dissimilarity between two data points $x_i$ and $x_{i'}$ across $J$ features is formally defined as:

$$D(x_i,x_{i'}) = \sum_{j=1}^J d_j(x_{ij}, x_{i'j})$$

For standard metric spaces, the squared Euclidean distance is utilized:

$$D(x_i,x_{i'}) = \sum_{j=1}^J (x_{ij} - x_{i'j})^2$$

2.2 Clustering Algorithms

• K-means Clustering: Segments data into $K$ clusters by minimizing the compactness, defined as the Within-Cluster Sum of Squares (WCSS):

  $$WCSS(C) = \sum_{k=1}^K \sum_{i \in C_k} ||x_i - \bar{x}_k||^2$$

  where $x_i$ is the $i$ th data point in cluster $C_k$ and $\bar{x}_k$ is the center of the $k$ th cluster.

• DBSCAN: Density-based clustering groups core, border, and noise points defined by a proximity radius $\epsilon$ and a threshold of minimum connecting points (minPts).

• Gaussian Mixture Models (GMM): Assumes data stems from a mixture of distributions. It employs soft clustering by maximizing posterior probabilities using the Expectation-Maximization (EM) algorithm to estimate weights $\pi_k$, means $\mu_k$, and covariance matrices $\Sigma_k$:

  $$p(x) = \sum_{k=1}^K \pi_k \mathcal{N}(x|\mu_k,\Sigma_k)$$

2.3 Cluster Validation

To prevent algorithms from imposing structure on random noise, internal validation techniques are necessary. The Silhouette Score computes the ratio of intra-cluster similarity ( $a$ ) against inter-cluster similarity ( $b$ ):

$$s(x) = \frac{b-a}{\max(a, b)}$$

The overall Silhouette Coefficient for a clustering partition $C$ containing $N$ samples averages the distinct scores ranging from $-1$ to $1$:

$$sil(C) = \frac{1}{N} \sum_{i=1}^N s(x_i)$$

Module 3: Supervised Learning & Optimization

Supervised learning models formulate mappings from input $x$ to true target labels $y$. Models $g(x; w)$ parameterize mapping functions to output predictions $\hat{y}$.

3.1 Loss Functions and Generalization Error

Loss functions measure prediction inaccuracies. The expected generalization risk over the true, unknown joint probability distribution $p(x, y)$ is:

$$R(g(.;w)) = \iint L(y, g(x;w)) \cdot p(x, y)dydx$$

Since $p(x, y)$ is unknowable, empirical risk minimization (ERM) approximations compute the loss against available data points $n$:

$$R_E(g(.;w), (X, y)) = \frac{1}{n} \sum_{i=1}^n L(y_i, g(x_i;w))$$

• Zero-one loss: Used for exact match classification.

  $$L_{zo}(y, g(x;w)) = \begin{cases} 0 & y = g(x;w) \\ 1 & y \neq g(x;w) \end{cases}$$

• Mean-Squared Error (MSE) / Quadratic Loss: Minimized iteratively or directly closed-form (in linear models) for regression tasks.

  $$L_q(y, g(x;w)) = (y - g(x;w))^2$$

3.2 Logistic Regression and Gradient Descent

Logistic classification applies a non-linear sigmoid activation function to map linear raw outputs $z = a + b \cdot x_i$ into probabilities constrained to $$:

$$\sigma(z) = \frac{1}{1 + e^{-z}}$$

Because the formulation has no closed-form analytical minimum, parameters are iteratively updated using Gradient Descent with a selected learning rate $\eta$:

$$\theta_{n+1} = \theta_n - \eta \frac{\partial L}{\partial \theta}$$

Module 4: Deep Learning & Convolutional Networks

4.1 Multi-class Softmax and Cross-Entropy

For $K > 2$ class architectures, the sigmoid function generalizes into the softmax equation to construct output probabilities:

$$p(y = i | x) = \sigma(z)_i = \frac{e^{z_i}}{\sum_{j=1}^K e^{z_j}}$$

Neural networks evaluate these probability distributions utilizing the Binary Cross Entropy (BCE) loss framework:

$$BCE = \frac{1}{n} \sum_{i=1}^n \left( -y_i \log(g(x_i; d, k)) - (1 - y_i) \log(1 - g(x_i; d, k)) \right)$$

4.2 Network Topologies

A fully connected linear layer containing weights $W$ and bias $b$ computes its outputs as:

$$z = b + Wx = b + \sum_{i=1}^M w_i x_i$$

Stacking multiple hidden layers $h_i(x) = b_i + W_i x$ successively collapses logically into a single affine transformation $h_n = b' + W'x$, proving the necessity of inter-layer non-linear activation functions (e.g., ReLU, Tanh) to scale deep architectures.

4.3 Convolutional Neural Networks (CNNs)

CNNs capture local spatial structures avoiding feed-forward parameter explosion by sharing kernel weights across a receptive field. Formally, the 2D mathematical convolution operation between matrix functions $h$ and $k$ slides the kernel across the input matrix:

$$(h * k)(a, b) = \sum_i \sum_j h(a+i, b+j)k(i, j)$$

Sub-operations in a CNN hierarchy include padding strategies to manipulate border overlaps and pooling topologies (Max or Average) to drastically down-sample activation maps prior to final flat-vector dense classifications.