Advanced Topic Modelling and Recommender Systems

Module 1: Fundamentals of Learning from User-Generated Data

User-generated data encompasses both explicitly created content (such as text, audio, images, and video) and implicitly created metadata (like system interactions, sensor data, and viewing behaviour). The core task of recommender systems (RS) is filtering this vast amount of data to satisfy specific user information needs. This contrasts with Information Retrieval (IR), where the user has a specific query, and Browsing, where the user has an undirected need. A recommender system proactively searches for and ranks relevant items based on a user's characteristics or past interactions.

Module 2: Collaborative Filtering (CF)

Collaborative filtering leverages historical user-item interactions to predict missing ratings, operating on the assumption that users with similar past preferences will share future preferences.

Memory-Based CF

Memory-based methods store all item ratings and compute predictions on the fly using a $k$-nearest neighbor approach.

• User-Based CF: Finds similar users to a target user $a$ using Pearson's correlation, which normalizes for user rating biases by subtracting the mean rating $\bar{r}_a$:

  $$sim(a,u) = \frac{\sum_{p \in P'} (r_{a,p} - \bar{r}_a)(r_{u,p} - \bar{r}_u)}{\sqrt{\sum_{p \in P'} (r_{a,p} - \bar{r}_a)^2} \sqrt{\sum_{p \in P'} (r_{u,p} - \bar{r}_u)^2}}$$

  The final rating is predicted as a weighted average of the neighbours' deviations from their mean.

• Item-Based CF: Finds similar items based on rating vectors across users. It generally scales better and employs the Adjusted Cosine Similarity to account for user rating biases.

Model-Based CF (Matrix Factorization)

Model-based approaches map users and items into a shared latent factor space of dimensionality $f$. A user $u$ is represented by a vector $w_u \in \mathbb{R}^f$, and an item $i$ by $h_i \in \mathbb{R}^f$. The predicted rating $r'_{u,i}$ is the inner product:

$$r'_{u,i} = w_u^T h_i$$

Singular Value Decomposition (SVD):
The rating matrix $R$ can be factorized into three matrices:

$$R = U \Sigma V^T$$

Objective Function & Gradient Descent:
To focus only on known ratings $L$ and avoid imputing missing values, the objective function minimizes the regularized squared error:

$$err = ||R - WH||_F^2 = \sum_{(u,i) \in L} (r_{ui} - w_u^T h_i)^2$$

The partial derivative for optimizing weight $w_{uK}$ using gradient descent is computed as:

$$\frac{\partial e_{uI}^2}{\partial w_{uK}} = -2(r_{uI} - r'_{uI})(h_{IK})$$

Incorporating Biases:
Pure interactions cannot explain full rating variance, so a global average $\mu$, item bias $b_i$, and user bias $b_u$ are added to the prediction: $r'_{u,i} = \mu + b_i + b_u + w_u^T h_i$. The regularized objective function becomes:

$$\min_{w*,h*,b*} \sum_{(u,i) \in L} (r_{u,i} - \mu - b_u - b_i - w_u^T h_i)^2 + \lambda(||w_u||^2 + ||h_i||^2 + b_u^2 + b_i^2)$$

Module 3: Evaluation Metrics

Evaluating RS performance can be conducted via offline testing, online A/B testing, and user studies.

Accuracy Metrics

• Mean Absolute Error (MAE): Treats small and large errors equally.

  $$MAE = \frac{1}{|T|} \sum_{r_{u,i} \in T} |r'_{u,i} - r_{u,i}|$$

• Root Mean Squared Error (RMSE): Disproportionately penalizes large prediction errors by squaring them.

  $$RMSE = \sqrt{\frac{1}{|T|} \sum_{(u,i) \in T} (r'_{u,i} - r_{u,i})^2}$$

Rank-Aware Metrics

• Mean Reciprocal Rank (MRR): The inverse of the position of the first relevant item.

  $$MRR = \frac{1}{\min_i \{i : L_u[i] \in T_u\}}$$

• Mean Average Precision (MAP): The arithmetic mean of the Average Precision (AP) values over all users $U$.

Beyond-Accuracy Metrics

• Novelty (Global): The inverse of overall item popularity $pop(i)$ in a result list of size $N$.

  $$Novelty = \frac{1}{|U|} \sum_{u \in U} \sum_{i \in L_u} \frac{-\log_2 pop(i)}{N}$$

• Item Coverage: The fraction of the catalog that the system retrieves or recommends.

• Intra-List Diversity: The average pairwise distance between all items in the recommendation list.

Module 4: Content-Based Filtering & Text Mining

Content-based filtering relies on computational features extracted from item content (e.g., text, audio, images) rather than interaction matrices.

Vector Space Model (VSM)

Documents are modeled as a Bag of Words (BoW) ignoring sequence but utilizing tokenization, stemming, and stop-word removal. Weights are often calculated using Term Frequency-Inverse Document Frequency (TF-IDF). The $ltc$ variant is mathematically defined as:

$$tfidf_{d,t} = (1 + \log_2 tf_{d,t}) \times \log_2 \left(\frac{N}{df_t}\right) \quad \text{if } tf_{d,t} > 0 \text{, otherwise } 0$$

Similarity between documents $d_i$ and $d_j$ is typically calculated using Cosine Similarity:

$$sim(d_i, d_j) = \cos \theta = \frac{d_i \cdot d_j}{|d_i| \times |d_j|}$$

Advanced Topic Modeling

Latent Semantic Analysis (LSA):
Employs SVD directly on the term-document matrix $X$ of dimensions $m \times n$ to uncover latent concepts. The matrix is decomposed into $X = T \cdot \Sigma \cdot D^T$, and the $k$ most important dimensions are retained for a minimum noise approximation:

$$X \approx X_k = T_k \cdot \Sigma_k \cdot D_k^T$$

Probabilistic Latent Semantic Analysis (pLSA):
Models terms and documents as mixtures of probabilistic latent topics $z \in Z$. The model maximizes the log-likelihood via Expectation-Maximization (EM):

$$\log L = \sum_{d \in D} \sum_{w \in W} tf_{d,w} \cdot \log \sum_{z \in Z} P(w|z) \cdot P(z|d)$$

Latent Dirichlet Allocation (LDA):
A generative probabilistic model where documents are modeled as Dirichlet/multinomial distributions over $K$ latent topics, and topics are distributions over vocabulary words $V$. The probability model is:

$$P(W,Z,\Theta,\Phi;\alpha,\beta) = \prod_{i=1}^K P(\phi_i, \beta) \prod_{j=1}^M P(\theta_j, \alpha) \prod_{t=1}^V P(z_{j,t}, \theta_j) \cdot P(w_{j,t}|\phi_{z_{j,t}})$$

Module 5: Graph, Hybrid, & Context-Aware Recommenders

Graph-Based Networks

Data is mapped into vertices $V$ and edges $E$ represented via an adjacency matrix $A$.

• Triadic Closure (Clustering Coefficient): Approximated via the 3rd moment of eigenvalues:

  $$C = \frac{3n_\Delta}{|E|} \approx \sum_i \lambda_i^3$$

• Katz Centrality: A radial, volume-based measure modeling diffusion:

  $$c_i^{KATZ} = e_i^T \sum_{j=1}^\infty \beta^j A^j \mathbf{1}$$

• Closeness Centrality: A length-based metric calculating average shortest distance $S$:

  $$c_i^{CLO} = \frac{1}{|V|} e_i^T S \mathbf{1}$$

Hybrid Recommenders

Combinations of different RS paradigms defined by Burke into three structural categories:

1. Parallelized Design: Separate recommenders output scores which are mixed, switched, or combined via a weighted sum subject to $\sum_{k=1}^h \beta_k = 1$.

2. Monolithic Design: A single system integrates multiple features in an early-fusion step (Feature Augmentation or Feature Combination). Content-Boosted CF creates a dense pseudo-ratings matrix using a harmonic mean self-weighting factor based on the number of ratings $n_u$:

   $$hm_{a,u} = \frac{2m_a m_u}{m_a + m_u}$$

3. Pipelined Design: Output of one model cascades into the next, refining or re-ranking candidate items.

Context-Aware and Psychology-Informed RS

Context dimensions include temporal/spatial environments and physiological states. Biases generated by these algorithms (like popularity bias) can be evaluated via delta popularity metrics:

$$\%\Delta\xi_{ui} = \frac{\xi_{pop}(R_{ui}) - \xi_{pop}(H_{ui})}{\xi_{pop}(H_{ui})} \cdot 100$$

Psychology-informed architectures incorporate models like OCEAN (Big Five) to capture user cognition, affect, and trait data directly into the mapping space.