The tuning guide draws one distinction and then builds on it: parameters are learned from data during training, while hyperparameters are the settings you choose before training. Finding good hyperparameters is what turns a mediocre model into a strong one. This workbook walks the full toolkit, from a hand-written loop through GridSearchCV and RandomizedSearchCV to coarse-to-fine refinement and tuning inside a pipeline. The rule that governs all of it appears in Example 3 and never lets go: tune with cross-validation on the training data, and keep the test set untouched until the very end.
1. Parameters versus hyperparameters
Before tuning anything, see the distinction in code. A model learns its parameters when you call fit; you set its hyperparameters when you construct it.
from sklearn.datasets import load_breast_cancerfrom sklearn.linear_model import LogisticRegressionX, y = load_breast_cancer(return_X_y=True)# C is a HYPERPARAMETER: we chose it before trainingmodel = LogisticRegression(C=1.0, max_iter=5000)model.fit(X, y)# coef_ are PARAMETERS: the model learned them from the dataprint("Hyperparameter C:", model.C)print("First 3 learned coefficients:", model.coef_[0][:3])
C is a dial you turned before fitting; the coefficients are what the fitting produced. Tuning is the search for the best settings of the dials, and everything in this workbook is a different way to run that search.
2. Manual search with a loop
The simplest tuning is a loop over candidate values, scoring each with cross-validation. It is crude, but for exploring one hyperparameter it is often all you need, and it makes the mechanics obvious.
from sklearn.datasets import load_breast_cancerfrom sklearn.neighbors import KNeighborsClassifierfrom sklearn.model_selection import cross_val_scoreX, y = load_breast_cancer(return_X_y=True)for k in [1, 3, 5, 11, 21, 41]: knn = KNeighborsClassifier(n_neighbors=k) scores = cross_val_score(knn, X, y, cv=5, scoring="accuracy") print(f"k={k:2d} cv accuracy = {scores.mean():.4f}")# k=1 0.9174 ... k=11 0.9350 ... k=41 0.9174 (a peak in the middle)
The scores rise to a peak around the middle values and then fall, which is the classic shape of a single hyperparameter: too small overfits, too large underfits, and the best value sits somewhere between. This loop is manual tuning in its entirety, and it motivates the automated tools that follow.
3. GridSearchCV: exhaustive and cross-validated
GridSearchCV tries every combination in a grid, scoring each with cross-validation, and remembers the best. This is the workhorse for small, discrete search spaces, and it enforces the golden rule: tuning happens on training data only.
from sklearn.datasets import load_breast_cancerfrom sklearn.ensemble import RandomForestClassifierfrom sklearn.model_selection import train_test_split, GridSearchCVX, y = load_breast_cancer(return_X_y=True)X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42)param_grid = { "n_estimators": [100, 300], "max_depth": [4, 8, None], "max_features": ["sqrt", "log2"],}grid = GridSearchCV( RandomForestClassifier(random_state=42), param_grid=param_grid, scoring="roc_auc", cv=5, n_jobs=-1, # use all CPU cores)grid.fit(X_train, y_train) # fit ONLY on training dataprint("Best params:", grid.best_params_)print("Best CV ROC-AUC:", round(grid.best_score_, 4))
The grid here is 2 × 3 × 2 = 12 combinations, each evaluated over 5 folds, so 60 fits in total. n_jobs=-1 runs them across every core, and the crucial line is grid.fit(X_train, ...): the search never sees the test set, so its choice cannot be contaminated by the data you will judge it on.
4. Reading the results
A fitted search is more than one number. best_params_, best_score_, and best_estimator_ give you the winner, and cv_results_holds the full leaderboard for inspection.
from sklearn.datasets import load_breast_cancerfrom sklearn.ensemble import RandomForestClassifierfrom sklearn.model_selection import train_test_split, GridSearchCVimport numpy as npX, y = load_breast_cancer(return_X_y=True)X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42)grid = GridSearchCV( RandomForestClassifier(random_state=42), param_grid={"n_estimators": [100, 300], "max_depth": [4, 8, None]}, scoring="roc_auc", cv=5, n_jobs=-1,)grid.fit(X_train, y_train)# best_estimator_ is the winning model, already refit on all training databest = grid.best_estimator_# rank the top 3 combinations from the full results tableresults = grid.cv_results_order = np.argsort(results["rank_test_score"])for i in order[:3]: print(round(results["mean_test_score"][i], 4), results["params"][i])
best_estimator_ is ready to use because refit=True is the default: after finding the best settings, GridSearchCV retrains one final model on all the training data. The cv_results_ table lets you see how close the runners-up were, which tells you whether the winner is a clear champion or a coin toss among near-equals.
5. Scoring the winner on the test set, correctly
Only now does the test set appear, exactly once, to get an honest estimate of performance. For ROC-AUC you must feed probabilities, not hard predictions, which is a pitfall the guide flags explicitly.
from sklearn.datasets import load_breast_cancerfrom sklearn.ensemble import RandomForestClassifierfrom sklearn.model_selection import train_test_split, GridSearchCVfrom sklearn.metrics import roc_auc_score, accuracy_scoreX, y = load_breast_cancer(return_X_y=True)X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42)grid = GridSearchCV( RandomForestClassifier(random_state=42), param_grid={"n_estimators": [100, 300], "max_depth": [4, 8, None]}, scoring="roc_auc", cv=5, n_jobs=-1,)grid.fit(X_train, y_train)best = grid.best_estimator_# ROC-AUC needs PROBABILITIES of the positive class, not predict()proba = best.predict_proba(X_test)[:, 1]preds = best.predict(X_test)print("Test ROC-AUC:", round(roc_auc_score(y_test, proba), 4))print("Test accuracy:", round(accuracy_score(y_test, preds), 4))
The predict_proba(X_test)[:, 1] slice is the detail people get wrong: roc_auc_score needs the model’s confidence, so passing plain predict() output silently produces a worse, incorrect score. Accuracy uses predict(); ROC-AUC uses probabilities. Match the metric to the right kind of output.
6. RandomizedSearchCV: sampling a large space
When the grid would be enormous, RandomizedSearchCV samples a fixed number of combinations instead of trying them all. You give it distributions to draw from and an n_iter budget.
from sklearn.datasets import load_breast_cancerfrom sklearn.ensemble import RandomForestClassifierfrom sklearn.model_selection import train_test_split, RandomizedSearchCVfrom scipy.stats import randintX, y = load_breast_cancer(return_X_y=True)X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42)param_distributions = { "n_estimators": randint(100, 500), # draw integers in a range "max_depth": randint(3, 15), "max_features": ["sqrt", "log2"],}search = RandomizedSearchCV( RandomForestClassifier(random_state=42), param_distributions=param_distributions, n_iter=20, # try 20 random combinations, not the full space scoring="roc_auc", cv=5, n_jobs=-1, random_state=42,)search.fit(X_train, y_train)print("Best params:", search.best_params_)print("Best CV ROC-AUC:", round(search.best_score_, 4))
Note param_distributions instead of param_grid, and the randint ranges instead of fixed lists. Twenty random draws often land close to the grid-search optimum at a fraction of the cost, which is why random search is the default choice once a space grows past a handful of combinations.
7. Grid versus random: the same budget compared
Why prefer random search for big spaces? Because with the same number of fits it explores more distinct values of each hyperparameter. This example pits them head to head under an equal budget.
from sklearn.datasets import load_breast_cancerfrom sklearn.ensemble import GradientBoostingClassifierfrom sklearn.model_selection import train_test_split, GridSearchCV, RandomizedSearchCVfrom scipy.stats import uniform, randintX, y = load_breast_cancer(return_X_y=True)X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42)# grid: 3 x 3 = 9 combinationsgrid = GridSearchCV( GradientBoostingClassifier(random_state=42), {"learning_rate": [0.01, 0.1, 0.2], "max_depth": [2, 3, 4]}, scoring="roc_auc", cv=5, n_jobs=-1,).fit(X_train, y_train)# random: 9 draws from continuous/int rangesrand = RandomizedSearchCV( GradientBoostingClassifier(random_state=42), {"learning_rate": uniform(0.01, 0.3), "max_depth": randint(2, 5)}, n_iter=9, scoring="roc_auc", cv=5, n_jobs=-1, random_state=42,).fit(X_train, y_train)print("Grid best:", round(grid.best_score_, 4), grid.best_params_)print("Random best:", round(rand.best_score_, 4), rand.best_params_)
Both run nine combinations, but the grid can only ever try three learning rates, while random search samples nine different ones from a continuous range. For hyperparameters that live on a continuum, like learning_rate, that wider coverage is exactly why random search tends to find better values for the same compute.
8. Coarse-to-fine refinement
The professional workflow is two stages: a wide, cheap random search to find the promising region, then a narrow, dense search to refine within it. You let stage one tell you where to zoom.
from sklearn.datasets import load_breast_cancerfrom sklearn.ensemble import RandomForestClassifierfrom sklearn.model_selection import train_test_split, RandomizedSearchCV, GridSearchCVfrom scipy.stats import randintX, y = load_breast_cancer(return_X_y=True)X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42)# STAGE 1: coarse, wide random searchcoarse = RandomizedSearchCV( RandomForestClassifier(random_state=42), {"n_estimators": randint(50, 600), "max_depth": randint(2, 20)}, n_iter=15, scoring="roc_auc", cv=5, n_jobs=-1, random_state=42,).fit(X_train, y_train)print("Coarse best:", coarse.best_params_) # e.g. n_estimators~300, max_depth~8# STAGE 2: fine grid, densely around the coarse winnerdepth = coarse.best_params_["max_depth"]fine = GridSearchCV( RandomForestClassifier(random_state=42), {"max_depth": [depth - 1, depth, depth + 1], "n_estimators": [250, 300, 350]}, scoring="roc_auc", cv=5, n_jobs=-1,).fit(X_train, y_train)print("Fine best:", fine.best_params_, "->", round(fine.best_score_, 4))
Stage one scans a huge range cheaply to locate the neighbourhood of good values; stage two searches densely inside that neighbourhood. This beats a single giant grid because you spend your fits where they matter instead of wasting them on regions that stage one already showed were hopeless.
9. Custom scoring and the danger of the wrong metric
The scoring argument decides what “best” means, and choosing it carelessly optimises for the wrong thing. On imbalanced data, accuracy misleads and a metric like F1 or ROC-AUC is the honest target.
from sklearn.datasets import make_classificationfrom sklearn.ensemble import RandomForestClassifierfrom sklearn.model_selection import train_test_split, GridSearchCV# a deliberately imbalanced dataset: ~10% positivesX, y = make_classification(n_samples=2000, weights=[0.9, 0.1], random_state=42)X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42)param_grid = {"max_depth": [3, 6, None], "n_estimators": [100, 300]}# same search, two different definitions of "best"for metric in ["accuracy", "f1"]: search = GridSearchCV( RandomForestClassifier(random_state=42), param_grid, scoring=metric, cv=5, n_jobs=-1, ).fit(X_train, y_train) print(f"scoring={metric:9s} -> best {metric} = {round(search.best_score_, 4)}, " f"params {search.best_params_}")
The two runs can pick different winners because they are optimising for different goals: accuracy can be high just by predicting the majority class, while F1 forces the model to actually find the rare positives. Decide what success means for your problem first, then set scoring to match, since the search will faithfully optimise whatever you tell it to.
10. Tuning inside a pipeline
The safe, real-world pattern wraps preprocessing and the model in a Pipeline, then tunes the whole thing at once. This prevents the subtle leakage of fitting a scaler on data that a cross-validation fold should not have seen.
from sklearn.datasets import load_breast_cancerfrom sklearn.preprocessing import StandardScalerfrom sklearn.svm import SVCfrom sklearn.pipeline import Pipelinefrom sklearn.model_selection import train_test_split, GridSearchCVX, y = load_breast_cancer(return_X_y=True)X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42)pipe = Pipeline([ ("scaler", StandardScaler()), # step 1: preprocessing ("svc", SVC(probability=True)), # step 2: the model])# name grid keys as step__hyperparameterparam_grid = { "svc__C": [0.1, 1, 10], "svc__gamma": ["scale", 0.01, 0.001],}grid = GridSearchCV(pipe, param_grid, scoring="roc_auc", cv=5, n_jobs=-1)grid.fit(X_train, y_train)print("Best params:", grid.best_params_)print("Best CV ROC-AUC:", round(grid.best_score_, 4))
The grid keys use the step__hyperparameter syntax, so svc__C means “the C of the step named svc.” The real prize is correctness: because the scaler lives inside the pipeline, it is refit on each training fold separately during cross-validation, so no validation fold ever influences the scaling. Tuning a bare model with a pre-scaled dataset leaks information; tuning the pipeline does not.
Work through these and you have the full article in practice: the parameter-versus-hyperparameter distinction, manual loops, GridSearchCV and RandomizedSearchCV, reading the results, correct test-set scoring, coarse-to-fine refinement, choosing the right metric, and tuning within a pipeline. The judgement the guide leaves you with is about matching strategy to situation: a quick loop to explore one dial, GridSearchCV for a small grid, RandomizedSearchCV for a large space, coarse-to-fine when fits are cheap, and always, always cross-validate on the training data so the test set stays a genuine final exam.
See you soon, Andrei.
[…] Hyperparameter Tuning in Python: 10 Code-Along Examples […]
[…] Hyperparameter Tuning in Python: 10 Code-Along Examples […]
[…] Hyperparameter Tuning in Python: 10 Code-Along Examples […]