Foundations of Statistical Science and AI Concepts: Master Study Guide

Module 1: Descriptive Statistics and Data Basics

Statistics is the science of learning from data by exploring, summarizing, modelling, estimating, and testing information. Descriptive statistics summarize raw data through tabular and graphical representations, and compute central tendency and variability measures.

Scales of Measurement

Data are categorized into multiple scales:

• Nominal: Unordered categorical data (e.g., gender, colour).
• Ordinal: Categories with a meaningful order but unquantifiable distances (e.g., school grades).
• Metric (Quantitative): Contains exact numeric values. Can be interval (arbitrary zero, e.g., dates) or ratio scale (absolute zero, enabling multiplication/division). Can be discrete (countable) or continuous (infinite precision).

Measures of Location (Central Tendency)

Location parameters summarize the "middle" or "average" of a dataset.

• Arithmetic Mean: The sum of values divided by population size. It is not robust to outliers or scale-invariant.
  $$\bar{x} = \frac{1}{N}\sum_{i=1}^N x_i$$

• Geometric Mean: Used for averaging growth factors (like interest rates). It is the N-th root of the product of N numbers.
  $$\bar{x}_{geo} = \sqrt[N]{\prod_{i=1}^N x_i}$$

• Harmonic Mean: Used when averaging rates or ratios defined in relation to a unit (e.g., speeds).
  $$\bar{x}_h = \frac{N}{\sum_{i=1}^N \frac{1}{x_i}}$$

• Weighted Arithmetic Mean: Applied when certain data points contribute more to the average.
  $$\bar{x}_{weighted} = \frac{\sum_{i=1}^N w_i x_i}{\sum_{i=1}^N w_i}$$

• Median: The middle value separating the higher and lower halves of ordered data, highly robust against outliers.

• For an odd $N$:
  $$\tilde{x}_{0.5} = x_{( \frac{N+1}{2} )}$$

• For an even $N$:
  $$\tilde{x}_{0.5} = \frac{1}{2}(x_{( \frac{N}{2} )} + x_{( \frac{N}{2} + 1 )})$$

• Mode: The most frequent value in a dataset.

Measures of Variability (Dispersion)

Variability defines the extent to which values differ from each other.

• Variance: The average squared deviation from the arithmetic mean. Empirical variance of a population is given by:
  $$Var(X) = \frac{1}{N}\sum_{i=1}^N (x_i - \bar{x})^2$$
  Using Steiner's theorem, this simplifies to:
  $$Var(X) = \frac{1}{N}\sum_{i=1}^N x_i^2 - \bar{x}^2$$
• Standard Deviation ( $SD(X)$ ): The square root of variance, reverting to the original unit of measurement.
  $$SD(X) = \sqrt{Var(X)}$$
• Coefficient of Variation: A standardized, unitless measure of relative variation compared to the mean.
  $$v = \frac{s}{\bar{x}} \cdot 100\%$$
• Kurtosis: Measures the "peakedness" or tailedness of a distribution. It computes the fourth central moment:
  $$w = \frac{1}{N}\sum_{i=1}^N \left(\frac{x_i - \bar{x}}{SD(X)}\right)^4 - 3$$

Module 2: Bivariate Data, Correlation, and Linear Regression

Bivariate data involves pairs of variables $(x_i, y_i)$ visualized via scatterplots.

Correlation

• Covariance: Measures the monotonic relationship between two variables.
  $$Cov(x, y) = \frac{1}{N}\sum_{i=1}^N (x_i - \bar{x}) \cdot (y_i - \bar{y}) = \frac{1}{N}\sum_{i=1}^N x_i y_i - \bar{x}\bar{y}$$
  For a sample, the estimator uses $n-1$:
  $$s_{xy} = \frac{1}{n-1}\sum_{i=1}^n (x_i - \bar{x}) \cdot (y_i - \bar{y})$$
• Bravais-Pearson Correlation Coefficient ( $r\_{xy}$ ): Normalizes covariance by the product of the standard deviations, returning a dimensionless value between -1 and +1.
  $$r_{xy} = \frac{Cov(x, y)}{SD(x) \cdot SD(y)}$$
• Spearman's Rank Correlation: Used for ordinal data or skewed metric data, computing Pearson correlation on the sorted ranks of the data to capture general monotonic trends.

Linear Regression

Linear regression models a response variable $Y$ based on an explanatory variable $X$.

• Regression Line: $\hat{y}_i = a + b \cdot x_i$.
• Residuals ( $e_i$ ): The deviation between observed values and predictions, $e_i = y_i - \hat{y}_i$.
• Goodness of Fit ( $R^2$ ): The coefficient of determination partitions the total variation:
  • Sum of Squares Total (SQT): $\sum_{i=1}^n (y_i - \bar{y})^2$
  • Sum of Squares Explained (SQE): $\sum_{i=1}^n (\hat{y}_i - \bar{y})^2$
  • Sum of Squares Residual (SQR): $\sum_{i=1}^n (y_i - \hat{y}_i)^2$
    $$R^2 = \frac{SQE}{SQT}$$

Module 3: Probability Theory and Combinatorics

Probability theory provides the mathematical foundation for inductive statistics.

Combinatorics (Counting Rules)

• Permutation without repetition: Arrangement of $n$ distinct objects.
  $$P_n = n!$$
• Permutation with repetition: Arrangement of $n$ objects containing duplicate groups.
  $$P_{n_1,n_2,...,n_r}^n = \frac{n!}{n_1! \cdot n_2! \cdot ... \cdot n_r!}$$
• Combination without repetition: Selection of $k$ objects from $n$ without order and without replacement.
  $$C_n^k = \frac{n!}{k!(n-k)!} = \binom{n}{k}$$
• Combination with repetition: Selection of $k$ objects from $n$ without order, but with replacement.
  $$RC_n^k = \binom{n+k-1}{k}$$
• Variation without repetition: Selection of $k$ objects from $n$ where order matters, without replacement.
  $$V_n^k = \frac{n!}{(n-k)!}$$
• Variation with repetition: Selection of $k$ objects from $n$ where order matters, with replacement.
  $$RV_n^k = n^k$$

Probability Rules and Bayes' Theorem

• Additive Rule: For the union of events.
  $$P(A \cup B) = P(A) + P(B) - P(A \cap B)$$
• Multiplicative Rule: For the intersection of events. If dependent:
  $$P(A \cap B) = P(A) \cdot P(B|A)$$
• Conditional Probability: $P(A|B) = \frac{P(A \cap B)}{P(B)}$.
• Law of Total Probability: Computes an overall probability via disjoint sub-events.
  $$P(B) = \sum_{i=1}^n P(A_i) \cdot P(B|A_i)$$
• Bayes' Theorem: Reverses conditional probabilities.
  $$P(A_i|B) = \frac{P(A_i) \cdot P(B|A_i)}{\sum_{j=1}^n P(A_j) \cdot P(B|A_j)}$$

Module 4: Random Variables and Probability Distributions

A random variable transforms experimental outcomes into real numbers ( $X: \Omega \rightarrow \mathbb{R}$ ).

Discrete vs. Continuous

• Discrete Random Variables: Utilize a Probability Mass Function (PMF), $f(x) = P(X = x)$. Expected value and variance are:
  $$\mu = E(X) = \sum_{i=1}^k x_i \cdot f(x_i)$$
  $$\sigma^2 = V(X) = \sum_{i=1}^k (x_i - \mu)^2 \cdot f(x_i)$$
• Continuous Random Variables: Utilize a Probability Density Function (PDF) where areas under the curve equal probabilities. The Cumulative Distribution Function (CDF) is:
  $$F(x) = \int_{-\infty}^x f(x) dx = P(X \leq x)$$
  Expected value and variance are:
  $$\mu = E(X) = \int_{-\infty}^{\infty} x \cdot f(x) dx$$
  $$\sigma^2 = V(X) = \int_{-\infty}^{\infty} (x - \mu)^2 \cdot f(x) dx$$

Key Distributions

1. Binomial Distribution: Models the number of successes in $n$ independent Bernoulli trials with replacement.
   $$f(x) = \binom{n}{x} \cdot p^x \cdot (1-p)^{n-x}$$
   $E(X) = n \cdot p$, $Var(X) = n \cdot p \cdot q$ (where $q = 1-p$ ).
2. Hypergeometric Distribution: Models successes in $n$ draws without replacement from population $N$ containing $M$ successes.
   $$f(x) = \frac{\binom{M}{x} \binom{N-M}{n-x}}{\binom{N}{n}}$$
3. Poisson Distribution: Models the frequency of rare events over a fixed interval ( $n \rightarrow \infty$, $p \rightarrow 0$ ).
   $$f(x) = \frac{\lambda^x}{x!} \cdot e^{-\lambda}$$
4. Normal (Gaussian) Distribution: Symmetric, bell-shaped continuous distribution defined by $\mu$ and $\sigma^2$.
   $$f(x) = \frac{1}{\sqrt{2\pi}\sigma} e^{-\frac{1}{2} \left(\frac{x-\mu}{\sigma}\right)^2}$$
5. Standard Normal Distribution: Mean $\mu = 0$, Variance $\sigma^2 = 1$. Computed via Z-transformation:
   $$z = \frac{x-\mu}{\sigma}$$
6. Chi-Square Distribution: Formed by the sum of $n$ squared independent Standard Normal variables.
   $$\sum_{i=1}^n Z_i^2 \sim \chi_n^2$$

Central Limit Theorem

The sum (or mean) of $n$ independent, identically distributed random variables converges to a normal distribution for sufficiently large $n$ ( $n > 30$ ).
$$Z_n = \frac{\sum_{i=1}^n X_i - n\mu}{\sqrt{n}\sigma} \sim N(0,1)$$

Module 5: Inductive Statistics and Hypothesis Testing

Inductive statistics estimates population parameters based on sample data. Point estimators should be unbiased (expected value converges to true parameter) and consistent (precision increases with sample size).

Confidence Intervals

A range bounding the true parameter with a specific probability.

• Mean, Known Variance:
  $$\mu \in \left[\bar{x} - u_{1-\alpha/2} \cdot \frac{\sigma}{\sqrt{n}} ; \bar{x} + u_{1-\alpha/2} \cdot \frac{\sigma}{\sqrt{n}}\right]$$
• Mean, Unknown Variance: Uses the t-distribution with $n-1$ degrees of freedom.
  $$\mu \in \left[\bar{x} - t_{n-1, 1-\alpha/2} \cdot \frac{s}{\sqrt{n}} ; \bar{x} + t_{n-1, 1-\alpha/2} \cdot \frac{s}{\sqrt{n}}\right]$$

Statistical Significance Testing

Hypothesis tests challenge a null hypothesis ( $H_0$ ) against an alternative hypothesis ( $H_1$ ).

• One-Sample Gauss Test (z-Test): Compares mean to reference $\mu_0$ with known variance.
  $$T(x) = \frac{\bar{x} - \mu_0}{\sigma / \sqrt{n}}$$
• Two-Sample t-Test (Independent): Compares two means assuming homogeneous variances.
  $$T(x, y) = \frac{\bar{x} - \bar{y}}{s_p \sqrt{\frac{n \cdot m}{n+m}}}$$
  Where pooled variance $s_p^2$ is:
  $$s_p^2 = \frac{(n-1)s_x^2 + (m-1)s_y^2}{n+m-2}$$
• F-Test for Homoscedasticity: Tests variance equality.
  $$F(x,y) = \frac{s_x^2}{s_y^2}$$
• Categorical Tests:
  • Chi-Square Goodness-of-Fit: Tests a distribution against reference frequencies $p_i$.

• McNemar's Change Test: Analyzes paired/dependent categorical data.