← All Google Cloud AI guidesFundamentals · 2.3, 3.2
ML fundamentals 2: bias-variance, regularization and evaluation metrics
The model half of ML fundamentals: underfitting vs overfitting, learning curves, regularization and ensembles, training-loop problems, validation strategies, and evaluation metrics in depth, including why PR AUC beats ROC AUC on imbalanced data, threshold selection, calibration, and regression and ranking metrics. Two calculators make the numbers concrete.
1. Bias, variance and generalization
A model's job is to generalize: do well on data it hasn't seen. Its error on new data comes from three sources:
| Source | Meaning | Symptom |
|---|---|---|
| Bias | The model is too simple, or its assumptions are wrong, so it misses real patterns | Underfitting: high error on training and validation data |
| Variance | The model is too sensitive to the particular training sample; it learns noise | Overfitting: low training error, much higher validation error |
| Irreducible error | Noise in the labels or missing information no model can recover | A floor that more modelling can't break |
Making a model more flexible usually lowers bias and raises variance; constraining it does the reverse. The goal is the capacity where validation error is lowest.
A student who memorizes past exam answers (high variance) aces practice papers and fails new questions. A student who learned only one rule of thumb (high bias) does badly on both. You want the one who learned the underlying ideas.
2. Diagnosing with learning curves
| What you see | Diagnosis | Try |
|---|---|---|
| Training and validation error both high, close together | High bias (underfitting) | A bigger or more flexible model, better features, less regularization, train longer |
| Training error low, validation error much higher | High variance (overfitting) | More data, augmentation, regularization (L2, dropout), early stopping, simpler model, fewer features |
| Validation loss falls, then rises while training loss keeps falling | Overfitting starts at the minimum | Early stopping at the best validation epoch; restore those weights |
| Gap closes as training data grows, validation still improving | More data will help | Collect or label more data |
| Both curves flat as data grows, error still high | More data won't help | Change the model or features |
| Validation better than training | Regularization active only in training (dropout), easier validation set, or leakage | Check the split and leakage before celebrating |
| Loss jumps around or diverges | Learning rate too high, bad data, exploding gradients | Lower the learning rate, clip gradients, check inputs |
3. Regularization and capacity control
| Technique | How it reduces overfitting | Notes |
|---|---|---|
| L2 (ridge, weight decay) | Penalizes large weights (sum of squares); spreads weight across features | Default choice; BigQuery ML l2_reg |
| L1 (lasso) | Penalizes absolute weights; drives some to exactly zero | Sparse models and feature selection; l1_reg |
| Elastic net | L1 + L2 | Correlated features plus sparsity |
| Dropout | Randomly zeroes neurons during training | Neural networks; inactive at inference |
| Early stopping | Stops when validation loss stops improving | Cheap and effective; BigQuery ML early_stop |
| Data augmentation | Adds realistic variations | Images, audio, text |
| Simpler model | Fewer layers, shallower trees (max_tree_depth), fewer features | Also cheaper to serve |
| Tree constraints | Minimum samples per leaf, subsampling rows and columns, learning rate (shrinkage) | Gradient boosting's main controls |
| Batch normalization | Stabilizes training; mild regularizing effect | Mainly an optimization aid |
Ensembles
| Method | Idea | Mainly reduces |
|---|---|---|
| Bagging (random forest) | Average many models trained on bootstrap samples | Variance |
| Boosting (XGBoost, gradient-boosted trees) | Each new model fixes the previous ones' errors | Bias (and variance with shrinkage and subsampling) |
| Stacking | A meta-model combines different models | Both; more complexity |
"Bagging calms variance, boosting beats bias."
4. Training-loop problems
| Problem | Sign | Fix |
|---|---|---|
| Learning rate too high | Loss oscillates, explodes or becomes NaN | Lower it; learning-rate schedules and warmup |
| Learning rate too low | Loss falls very slowly | Raise it; adaptive optimizers (Adam) |
| Vanishing gradients | Early layers stop learning in deep networks | ReLU-family activations, residual connections, batch norm, good initialization |
| Exploding gradients | Huge updates, NaN loss (common in RNNs) | Gradient clipping, lower learning rate, normalization |
| Dead ReLUs | Many neurons always output zero | Leaky ReLU, lower learning rate |
| NaN loss | Bad inputs (NaN, inf), log(0), division by zero | Validate data, add epsilons, clip |
| Batch size effects | Large batches: faster per epoch, may generalize worse; small: noisier, slower | Scale the learning rate with batch size; tune both |
5. Validation strategies
| Strategy | Use when |
|---|---|
| Hold-out validation | Plenty of data; fast |
| k-fold cross-validation | Small or medium data; more reliable estimates (train k times, each fold validates once) |
| Stratified k-fold | Classification, especially imbalanced |
| Group k-fold | Many rows per entity |
| Time-series (rolling or expanding window) validation | Forecasting; never shuffle; also called backtesting |
| Nested cross-validation | Unbiased estimate when also tuning hyperparameters |
Every choice made by looking at a dataset (features, hyperparameters, thresholds) "uses up" that dataset. Make those choices on validation data; report the final number on untouched test data.
6. Classification metrics
| Predicted positive | Predicted negative | |
|---|---|---|
| Actually positive | True positive (TP) | False negative (FN), a miss |
| Actually negative | False positive (FP), a false alarm | True negative (TN) |
| Metric | Formula | Answers | Prefer when |
|---|---|---|---|
| Precision | TP / (TP + FP) | When it says positive, how often is it right? | False positives are costly: blocking good customers, spam filtering real mail |
| Recall (sensitivity, true positive rate) | TP / (TP + FN) | Of all real positives, how many did it find? | Misses are costly: fraud, disease screening, safety defects |
| Specificity | TN / (TN + FP) | Of real negatives, how many were cleared? | Medical testing pairs it with sensitivity |
| False positive rate (FPR) | FP / (FP + TN) = 1 − specificity | How often negatives are flagged | The x-axis of the ROC curve |
| F1 | 2PR / (P + R) | Balance of precision and recall | One number when both matter; F-beta weights recall β times as much |
| Accuracy | (TP + TN) / all | Fraction correct | Only for roughly balanced classes with similar error costs |
| Log loss (cross-entropy) | −mean(y·log p + (1−y)·log(1−p)) | How good the probabilities are | Probabilities are used directly; punishes confident mistakes |
Multi-class averaging
- Macro average: mean of per-class metrics; every class counts equally (good when rare classes matter).
- Micro average: pool all decisions; dominated by frequent classes (equals accuracy for single-label problems).
- Weighted average: per-class metrics weighted by class frequency.
7. ROC AUC vs PR AUC: why imbalance changes the answer
Both curves sweep the decision threshold from high to low.
- ROC curve: true positive rate (recall) against false positive rate. ROC AUC is the probability that a random positive is scored above a random negative. 0.5 is random; 1.0 is perfect.
- PR curve: precision against recall. PR AUC (average precision) is the area under it. A random model scores about the positive rate (0.01 at 1% positives), not 0.5.
Worked example
1,000,000 transactions, 1% fraud (10,000 positives). A model at some threshold has recall 80% and FPR 1%.
| Quantity | Value |
|---|---|
| True positives | 0.80 × 10,000 = 8,000 |
| False positives | 0.01 × 990,000 = 9,900 |
| Precision | 8,000 / 17,900 ≈ 45% |
On the ROC curve, (FPR 1%, TPR 80%) looks excellent. But more than half the alerts are false. If fraud were 0.1%, the same TPR and FPR would give precision ≈ 7%. ROC doesn't move when prevalence changes, because both of its rates are computed within one class each; precision mixes the two classes, so PR curves show what an analyst reviewing alerts experiences.
Imbalanced data: use PR AUC (and precision at a required recall, or recall at a required precision). ROC AUC is fine for balanced classes and for comparing ranking quality, but on rare-event problems it can look great while the model is unusable.
PR AUC values aren't comparable across datasets with different positive rates; the baseline moves with prevalence. Compare models on the same test set.
8. Prevalence calculator: ROC point to precision
Pick a recall (true positive rate) and false positive rate, then change the positive rate to see what precision users would actually experience.
9. Choosing the threshold
Models output scores; a threshold turns scores into decisions. The default 0.5 is arbitrary. Choose the threshold on validation data from what each error costs.
- Price each error: a missed fraud costs the transaction amount; a false alarm costs review time and customer friction.
- For each candidate threshold, compute expected cost = FN × costFN + FP × costFP.
- Pick the minimum, or the threshold that meets a hard constraint (recall ≥ 95%, or at most 500 alerts per day for the review team).
- Raising the threshold: fewer positives, usually higher precision and lower recall.
- BigQuery ML:
ML.ROC_CURVEandML.CONFUSION_MATRIX(MODEL m, (SELECT ...), STRUCT(0.3 AS threshold)); predictions accept a threshold too. - Thresholds can differ per segment only if policy and fairness review allow it.
10. Confusion-matrix calculator
Enter counts at your current threshold to get every metric and which way to move the threshold.
11. Calibration
A model is calibrated if, among cases it scores 0.8, about 80% are positive. Ranking (AUC) can be excellent while probabilities are badly off.
| Topic | Detail |
|---|---|
| Why it matters | Expected-cost decisions, risk scores shown to people, combining models, pricing |
| What breaks it | Class weights, resampling, boosting, overconfident neural networks, distribution shift |
| How to check | Reliability diagram (predicted vs observed rate per bin), Brier score, log loss |
| How to fix | Platt scaling (logistic fit on scores), isotonic regression (more flexible, needs more data), temperature scaling for neural networks; fit on validation data |
12. Regression and forecasting metrics
| Metric | Meaning | Prefer when |
|---|---|---|
| MAE (mean absolute error) | Average absolute miss, in the target's units | Robust to outliers; easy to explain |
| RMSE (root mean squared error) | Squares errors, so large misses count much more | Big errors are disproportionately bad |
| MSE | RMSE squared; common training loss | Optimization |
| Huber loss | Squared for small errors, absolute for large | Training with some outliers |
| R² | Share of variance explained (1 is perfect, 0 equals predicting the mean) | Comparing to a mean baseline |
| MAPE | Mean absolute percentage error | Scale-free comparison; breaks near zero actuals, penalizes over-forecasts more |
| WAPE / sMAPE | Weighted or symmetric percentage errors | Forecasting with intermittent or near-zero demand |
| RMSLE | RMSE on log values | Relative errors matter; under-prediction worse than over |
| Quantile (pinball) loss, interval coverage | Quality of prediction intervals | Inventory and capacity planning |
13. Ranking, recommendation and business metrics
| Metric | Meaning |
|---|---|
| Precision@k, recall@k | Quality of the top k results |
| MAP (mean average precision) | Rewards putting relevant items early, averaged over queries |
| NDCG (normalized discounted cumulative gain) | Graded relevance with a position discount |
| MRR (mean reciprocal rank) | 1 / rank of the first relevant result |
| Coverage, diversity, novelty | Whether recommendations go beyond the popular few |
| Online metrics | Click-through rate, conversion, revenue per session, retention: measured by A/B tests |
Offline metrics choose candidates; online experiments decide launches. A model can improve NDCG offline and still lose an A/B test on revenue.
14. "What does this training curve mean?" chooser
Pick an item to see what it means, why, and the tempting wrong choice.
Pick one above.
15. Scenarios
Exam-style situations: how to diagnose each one, the likely cause and the fix.
16. Searchable reference
| Item | Meaning |
|---|
17. Flashcards (click to flip)
18. Common gotchas in one list
- Both errors high = bias; big gap = variance. The fixes are opposite.
- More data fixes variance, not bias.
- Regularization and early stopping fight overfitting; they make underfitting worse.
- PR AUC for imbalanced data. ROC AUC ignores prevalence and can look great while precision is terrible.
- PR AUC's random baseline is the positive rate, not 0.5.
- Never keep the 0.5 threshold by default; pick it from error costs on validation data.
- Precision when false positives hurt, recall when misses hurt.
- Class weights and resampling break calibration; recalibrate before using probabilities.
- Tuning on the test set makes the test score optimistic.
- Time series: no shuffling; backtest with rolling windows.
- RMSE punishes big misses; MAE is robust; MAPE breaks near zero.
- Macro averages expose rare-class failure that accuracy hides.
- Offline metrics pick candidates; A/B tests decide launches.
- Validation better than training: check dropout effects and leakage before celebrating.
19. Practice quiz (24 questions)
Score: 0 / 0