Classical machine learning

Before deep learning is on the table, classical ML solves most contest tabular problems faster, with fewer bugs, and with smaller compute. This page sweeps the scikit-learn families you must own: regression, classification, ensembles, cross-validation, clustering, dimensionality reduction.

The contest workflow

1. Load & inspect. pd.read_csv, .info(), .describe(), scan for NaNs, look at value distributions.
2. Train/val/test split. Stratified for classification; time-based for time series. Never mix.
3. Baseline model. Logistic regression or random forest with default params. Whatever you build later must beat this.
4. Feature engineering. Numerical: standardize / log-transform. Categorical: one-hot or target encode. Date: extract year / month / day-of-week.
5. Model search. Try 2–3 model families, tune each with cross-validation.
6. Ensemble. Average top models. Usually +1–3% on the leaderboard.
7. Reproduce. Fix random seeds, save the trained model, log the val score.

1. Regression

Concept

Linear regression fits ŷ = Xw + b by minimising MSE ‖y − Xw − b‖². Closed form: w = (XᵀX)⁻¹ Xᵀy (when XᵀX is invertible). Adding L2 (ridge) regularisation gives (XᵀX + αI)⁻¹ Xᵀy — it shrinks weights and is numerically stable even when features are correlated. L1 (lasso) drives many weights to exactly zero, so it performs feature selection; the cost is a non-smooth objective solved by coordinate descent. Elastic Net blends L1 + L2 with a mixing parameter.

Pick the metric to match the cost: MAE for "I care equally about every dollar", RMSE when large errors hurt disproportionately, R² for "fraction of variance explained relative to predicting the mean".

Worked example — ridge vs. lasso on the diabetes dataset

from sklearn.datasets import load_diabetes
from sklearn.model_selection import train_test_split
from sklearn.linear_model import Ridge, Lasso
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
from sklearn.metrics import mean_squared_error, r2_score
import numpy as np

X, y = load_diabetes(return_X_y=True)
Xtr, Xva, ytr, yva = train_test_split(X, y, test_size=0.2, random_state=0)

for name, model in [("ridge", Ridge(alpha=1.0)),
                    ("lasso", Lasso(alpha=0.1, max_iter=10000))]:
    pipe = Pipeline([("s", StandardScaler()), ("m", model)]).fit(Xtr, ytr)
    pred = pipe.predict(Xva)
    print(name,
          "RMSE", round(np.sqrt(mean_squared_error(yva, pred)), 2),
          "R^2",  round(r2_score(yva, pred), 3),
          "nonzero weights", int(np.sum(pipe.named_steps["m"].coef_ != 0)))

Drills

2. Classification

Concept

Logistic regression: p = σ(wᵀx + b), train by minimising binary cross-entropy −Σ yᵢ log pᵢ + (1−yᵢ) log(1−pᵢ). Linear decision boundary, calibrated probabilities, hard to overfit. kNN: predict the majority label among the k nearest training points; no training, but inference cost O(n) per query and demands feature scaling. SVM: maximises margin; with the RBF kernel, fits non-linear boundaries via the kernel trick but scales poorly past ~10k points. Trees are non-linear, scale-free, handle missingness, but a single tree overfits — bag many (random forest) or boost them sequentially (gradient boosting).

Metrics: accuracy is misleading on imbalanced data. Precision = TP/(TP+FP) is "of the ones I flagged, how many were real". Recall = TP/(TP+FN) is "of the real positives, how many did I catch". F1 is the harmonic mean — useful when you want a single number that punishes either being zero. ROC-AUC ranks positives above negatives, threshold- independent.

Worked example — logistic regression with a class-imbalance fix

from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import classification_report, roc_auc_score

X, y = make_classification(n_samples=5000, n_features=20,
                           weights=[0.97, 0.03], random_state=0)
Xtr, Xva, ytr, yva = train_test_split(X, y, stratify=y, test_size=0.25, random_state=0)

clf = Pipeline([
    ("s", StandardScaler()),
    ("m", LogisticRegression(max_iter=1000, class_weight="balanced")),
]).fit(Xtr, ytr)

pred  = clf.predict(Xva)
proba = clf.predict_proba(Xva)[:, 1]
print(classification_report(yva, pred, digits=3))
print("ROC-AUC:", round(roc_auc_score(yva, proba), 3))

Drills

3. Trees & ensembles

Concept

A decision tree splits the feature space along axis-aligned cuts that maximally reduce impurity (Gini or entropy for classification, variance for regression). Single trees have low bias but huge variance. Bagging trains T trees on bootstrap samples and averages; random forests further decorrelate by sampling features at each split. Boosting trains trees sequentially, each fit to the residuals (gradient boosting) or weighted errors (AdaBoost) of the previous ensemble. Gradient boosting on histograms (LightGBM, XGBoost, scikit-learn's HistGradientBoosting*) is the state of the art for tabular data.

Worked example — Random forest vs. gradient boosting

from sklearn.datasets import fetch_california_housing
from sklearn.model_selection import cross_val_score, KFold
from sklearn.ensemble import RandomForestRegressor, HistGradientBoostingRegressor
import numpy as np

X, y = fetch_california_housing(return_X_y=True)
cv = KFold(n_splits=5, shuffle=True, random_state=0)

for name, model in [("RF", RandomForestRegressor(n_estimators=300, random_state=0, n_jobs=-1)),
                    ("HGB", HistGradientBoostingRegressor(max_iter=500, learning_rate=0.05,
                                                          random_state=0))]:
    scores = cross_val_score(model, X, y, cv=cv,
                             scoring="neg_root_mean_squared_error", n_jobs=-1)
    print(name, "RMSE", -scores.mean().round(3), "±", scores.std().round(3))

Drills

4. Cross-validation & bias / variance

Concept

A model's expected error decomposes as bias² (systematic mismatch between model and truth) + variance (sensitivity to the particular training sample) + irreducible noise. Underfitting = high bias; overfitting = high variance. Regularisation, more data, and simpler models reduce variance; richer model families and more features reduce bias.

K-fold cross-validation estimates expected val error: split into k folds, train on k−1, evaluate on the held-out fold, repeat, report mean ± std. Use stratified K-fold for classification (preserves class balance) and TimeSeriesSplit for time-ordered data (never train on the future).

Worked example — bias/variance with a learning curve

from sklearn.model_selection import learning_curve, StratifiedKFold
from sklearn.linear_model import LogisticRegression
from sklearn.datasets import load_breast_cancer
import numpy as np, matplotlib.pyplot as plt

X, y = load_breast_cancer(return_X_y=True)
sizes, tr, va = learning_curve(
    LogisticRegression(max_iter=2000), X, y,
    cv=StratifiedKFold(5, shuffle=True, random_state=0),
    train_sizes=np.linspace(0.1, 1.0, 8), scoring="accuracy")

plt.plot(sizes, tr.mean(1), label="train")
plt.plot(sizes, va.mean(1), label="val")
plt.xlabel("training set size"); plt.ylabel("accuracy"); plt.legend()
# wide gap -> variance problem; both low and flat -> bias problem

Drills

5. Unsupervised learning

Concept

k-means alternates assigning each point to its nearest centroid and updating each centroid as the mean of its cluster. It minimises within-cluster sum of squares but converges to local minima — use k-means++ initialisation and pick k with the elbow method or silhouette score. Hierarchical (agglomerative) clustering builds a tree of merges and lets you cut at any level; useful when k is unknown and the dataset is small (cost is O(n²) or worse).

PCA projects data onto the top-k eigenvectors of the covariance matrix — the directions of greatest variance. Centred input is mandatory; standardise if features are on different scales. t-SNE and UMAP are nonlinear; they are for visualization, not for downstream features (distances aren't globally meaningful).

Worked example — PCA + k-means on the iris dataset

from sklearn.datasets import load_iris
from sklearn.preprocessing import StandardScaler
from sklearn.decomposition import PCA
from sklearn.cluster import KMeans
from sklearn.metrics import adjusted_rand_score
import numpy as np

X, y = load_iris(return_X_y=True)
Xs = StandardScaler().fit_transform(X)

pca = PCA(n_components=2).fit(Xs)
Xp  = pca.transform(Xs)
print("variance explained:", pca.explained_variance_ratio_)  # ~[0.73, 0.23]

km = KMeans(n_clusters=3, n_init=10, random_state=0).fit(Xp)
print("ARI vs. true species:", adjusted_rand_score(y, km.labels_))

Drills

Pitfalls cheat-sheet

• Data leakage. Scaler/feature stats fit on full data; targets-derived features; future info bleeding into past rows.
• Overfitting the val set. Hundreds of hyperparameter sweeps against one split.
• Class imbalance. A 99%-accurate model on 99/1 data may be predicting "majority" every time. Stratify, use class weights, or resample.
• Forgotten seeds. Two identical runs producing different scores wastes hours.
• Forgetting to refit on full training data after cross-validation. CV gives you the score estimate; for the final submission, refit on all available training data.

Checkpoint — answer out loud

• Can you explain bias vs. variance using one sentence and one example each?
• Can you set up a stratified 5-fold CV pipeline that includes scaling without leakage?
• Can you compute precision, recall, and F1 from a confusion matrix?
• Can you state the difference between bagging and boosting, and pick when to use which?
• Can you describe — in geometric terms — why lasso zeroes weights and ridge does not?

Next step

When tabular gradient boosting plateaus, you reach for neural nets — head to Deep learning with PyTorch. Test your ML reasoning on bias–variance, CV bias, information gain, and the SVM dual at the Round 2 theory drills.