--- title: "Regularization and Constraints" output: rmarkdown::html_vignette vignette: > %\VignetteIndexEntry{Regularization and Constraints} %\VignetteEngine{knitr::rmarkdown} %\VignetteEncoding{UTF-8} --- ```{r setup, include = FALSE} knitr::opts_chunk$set( dpi = 72, fig.retina = 1, # keep the built vignettes small (CRAN size limit) collapse = TRUE, comment = "#>", fig.width = 7, fig.height = 4.5, message = FALSE, warning = FALSE ) library(RcppML) # CRAN allows at most two cores options(RcppML.threads = 2) # Chunks run only when NOT_CRAN is "true" (set when the package tarball and # the pkgdown site are built) and the suggested packages are installed. # CRAN's checks then rebuild the text without refitting the models. run_chunks <- identical(Sys.getenv("NOT_CRAN"), "true") && all(vapply(c("ggplot2"), requireNamespace, logical(1), quietly = TRUE)) knitr::opts_chunk$set(eval = run_chunks) if (run_chunks) { library(ggplot2) } # Helper: rescale a numeric vector to [0, 1] rescale01 <- function(x) { rng <- range(x) if (diff(rng) > 0) (x - rng[1]) / diff(rng) else rep(0.5, length(x)) } ``` ## Why Regularize? Unregularized NMF often produces dense factors where every feature loads on every component. The result is hard to interpret — factors overlap extensively and capture redundant structure. Regularization gives you control: **sparse** factors (few active features per component), **decorrelated** factors (minimal redundancy between components), or **smooth** factors (respecting spatial or network structure in the data). RcppML provides six distinct penalty mechanisms, all applied independently to W and H. ## API Reference ### Sparsity Penalties - `L1 = c(w_penalty, h_penalty)` — LASSO penalty, range [0, 1). Drives small coefficients to exactly zero, producing element-wise sparsity. The penalty is an absolute shift of the right-hand side of each solve, applied the same way on every fit path (sparse, dense, masked, cross-validated and non-Gaussian). Because the factors are scaled to unit sum by the diagonal `d`, the same value has a comparable effect across datasets. - `L2 = c(w_penalty, h_penalty)` — Ridge penalty, range ≥ 0. Shrinks large coefficients toward zero without producing exact zeros. - `L21 = c(w_penalty, h_penalty)` — Group sparsity, range ≥ 0. A group is one feature for W (a row of W, its loadings on all factors) and one sample for H (a column of H). The penalty shrinks whole groups together, so it removes a feature or a sample from the model entirely instead of zeroing single entries. ### Structural Penalties - `angular = c(w_penalty, h_penalty)` — Orthogonality penalty, range ≥ 0. Penalizes correlation between factor pairs, encouraging decorrelated components. - `graph_W`, `graph_H` — Sparse, symmetric graph Laplacian matrices (`dgCMatrix`), $L = D - A$ for an undirected graph with adjacency $A$ and degree matrix $D$: features × features for `graph_W`, samples × samples for `graph_H`. Penalizes differences between connected nodes, enforcing spatial or network smoothness. Pass the Laplacian, not the adjacency matrix. - `graph_lambda = c(w_strength, h_strength)` — Controls the strength of graph regularization. ### Box Constraints - `upper_bound = c(w_bound, h_bound)` — Maximum coefficient value (0 = no bound). The bound is a constraint of the least-squares solve itself, like non-negativity, so the returned coefficients are the best fit within the box. Useful for bounded data such as probabilities or normalized scores. - `nonneg = c(TRUE, TRUE)` — Nonnegativity toggle. Set `FALSE` to allow negative coefficients (semi-NMF). ### Parameter Conventions All penalty parameters accept either a single value (applied to both W and H) or a length-2 vector `c(w, h)` for independent control. Zero means no penalty (the default for all). Each penalty modifies the NNLS subproblem: L1 subtracts from the RHS (soft thresholding), L2 adds to the Gram diagonal (Tikhonov), angular applies a post-solve projection that subtracts correlated components between factors (decorrelation), the graph Laplacian couples the solutions of connected nodes (smoothness), and the upper bound and non-negativity are box constraints of the solve. The penalties are honored on every fit path, including masked data, cross-validation and the non-Gaussian losses. ## Theory NMF solves $A \approx W \cdot \text{diag}(d) \cdot H$ by alternating NNLS updates. Each penalty extends the per-column subproblem: $$\min_{h \geq 0} \|Gh - b\|^2 + \lambda_1 \|h\|_1 + \lambda_2 \|h\|^2_2$$ The full penalized NMF objective is: $$\mathcal{L} = \|A - WDH\|_F^2 + \lambda_{L1} \sum_{ij} |W_{ij}| + \lambda_{L2} \sum_{ij} W_{ij}^2 + \lambda_{\text{ang}} \sum_{i < j} \cos(w_i, w_j) + \lambda_G \, \text{tr}(H L H^\top)$$ Each penalty modifies the NNLS subproblem: - **L1 (LASSO)**: $\lambda_1 \|h\|_1$ — soft-thresholds the RHS, driving small coefficients to exactly zero. Ideal for sparse factors. - **L2 (Ridge)**: $\lambda_2 \|h\|_2^2$ — adds $\lambda_2 I$ to the Gram diagonal (Tikhonov regularization). Shrinks coefficients smoothly. - **L21 (Group LASSO)**: $\lambda_{21} \sum_g \|w_{g\cdot}\|_2$ for W, with one group per feature (row of W), and $\lambda_{21} \sum_g \|h_{\cdot g}\|_2$ for H, with one group per sample (column of H). Useful for feature selection. - **Angular**: $\lambda_a \sum_{i