Learning Algorithms

CSI 4106 - Fall 2026

Marcel Turcotte

Version: Sep 15, 2026 16:04

Preamble

Message of the Day

Learning outcomes

  • Differentiate between model, objective, and optimizer in learning algorithms.
  • Describe decision trees and apply entropy to select a split.
  • Explain KNN for classification and regression, including uniform and distance-weighted prediction.
  • Interpret decision boundaries and the concept of linear separability.

Decision Tree

Interpretable

What is a Decision Tree?

  • A decision tree is a rooted, hierarchical structure used for classification and regression tasks.
  • Each internal node is a decision node that performs a binary test on a particular feature (j), such as evaluating whether the number of connections at a school surpasses a specified threshold.
  • Each leaf produces a prediction: a class label or class probabilities for classification, or a numerical value for regression.

Classifying New Instances (Inference)

  • Begin at the root node of the decision tree. Proceed by answering a sequence of binary questions until a leaf node is reached. The label associated with this leaf denotes the classification of the instance.
  • Alternatively, some algorithms may store a probability distribution at the leaf, representing the fraction of training samples corresponding to each class k, across all possible classes k.

Decision Boundary

Palmer Penguins Dataset

# Loading our dataset

try:
  from palmerpenguins import load_penguins
except:
  ! pip install palmerpenguins
  from palmerpenguins import load_penguins

penguins = load_penguins()

# Pairplot using seaborn

import matplotlib.pyplot as plt
import seaborn as sns

sns.pairplot(penguins, hue='species', markers=["o", "s", "D"])
plt.suptitle("Pairwise Scatter Plots of Penguins Features")
plt.show()

Palmer Penguins Dataset

Binary Classification Problem

  • Several scatter plots reveal a distinct clustering of Gentoo instances.
  • To illustrate our next example, we propose a binary classification model: Gentoo versus non-Gentoo.
  • Our analysis will concentrate on two key features: body mass and bill depth.

Definition

A decision boundary is a “boundary” that partitions the underlying feature space into regions corresponding to different class labels.

Decision Boundary

The decision boundary between these attributes can be represented as a line.

Code 
# Import necessary libraries
import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split

try:
  from palmerpenguins import load_penguins
except:
  ! pip install palmerpenguins
  from palmerpenguins import load_penguins

# Load the Palmer Penguins dataset
df = load_penguins()

# Preserve only the necessary features: 'bill_depth_mm' and 'body_mass_g'
features = ['bill_depth_mm', 'body_mass_g']
df = df[features + ['species']]

# Drop rows with missing values
df.dropna(inplace=True)

# Create a binary problem: 'Gentoo' vs 'Not Gentoo'
df['species_binary'] = df['species'].apply(lambda x: 1 if x == 'Gentoo' else 0)

# Define feature matrix X and target vector y
X = df[features].values
y = df['species_binary'].values

# Split the data into training and testing sets
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)

# Function to plot initial scatter of data
def plot_scatter(X, y):
    plt.figure(figsize=(9, 5))
    plt.scatter(X[y == 1, 0], X[y == 1, 1], color='orange', edgecolors='k', marker='o', label='Gentoo')
    plt.scatter(X[y == 0, 0], X[y == 0, 1], color='blue', edgecolors='k', marker='o', label='Not Gentoo')
    plt.xlabel('Bill Depth (mm)')
    plt.ylabel('Body Mass (g)')
    plt.title('Scatter Plot of Bill Depth vs. Body Mass')
    plt.legend()
    plt.show()
    
# Plot the initial scatter plot
plot_scatter(X_train, y_train)

Decision Boundary

Decision Boundary

The decision boundary between these attributes can be represented as a line.

Code 
# Train a logistic regression model
model = LogisticRegression()
model.fit(X_train, y_train)

# Function to plot decision boundary
def plot_decision_boundary(X, y, model):
    x_min, x_max = X[:, 0].min() - 1, X[:, 0].max() + 1
    y_min, y_max = X[:, 1].min() - 1, X[:, 1].max() + 1
    xx, yy = np.meshgrid(
        np.linspace(x_min, x_max, 300),
        np.linspace(y_min, y_max, 300)
    )
    Z = model.predict(np.c_[xx.ravel(), yy.ravel()])
    Z = Z.reshape(xx.shape)
    
    plt.figure(figsize=(9, 5))
    plt.contourf(xx, yy, Z, alpha=0.3, cmap='RdYlBu')
    plt.scatter(X[y == 1, 0], X[y == 1, 1], color='orange', edgecolors='k', marker='o', label='Gentoo')
    plt.scatter(X[y == 0, 0], X[y == 0, 1], color='blue', edgecolors='k', marker='o', label='Not Gentoo')
    plt.xlabel('Bill Depth (mm)')
    plt.ylabel('Body Mass (g)')
    plt.title('Logistic Regression Decision Boundary')
    plt.legend()
    plt.show()

# Plot the decision boundary on the training set
plot_decision_boundary(X_train, y_train, model)

Decision Boundary

Definition

We say that the data is linearly separable when two classes of data can be perfectly separated by a single linear boundary, such as a line in two-dimensional space or a hyperplane in higher dimensions.

Simple Decision Boundary

(a) training data, (b) quadratic curve, and (c) linear function.

Complex Decision Boundary

Decision trees are capable of generating irregular and non-linear decision boundaries.

Definition (revised)

A decision boundary is a hypersurface that partitions the underlying feature space into regions corresponding to different class labels.

Decision Tree (contd)

Constructing a Decision Tree

  • How to construct (learnt) a decision tree?

  • Are there some trees that are “better” than others?

  • Is it feasible to construct an optimal decision tree with computational efficiency?

Optimality

  • Let X = \{x_1, \ldots, x_n\} be a finite set of objects.
  • Let \mathcal{T} = \{T_1, \ldots, T_t\} be a finite set of tests.
  • For each object and test, we have:
    • T_i(x_j) is either true or false.
  • An optimal tree is one that completely identifies all the objects in X and |T| is minimum.

Constructing a Decision Tree

  • Iterative development: Initiate with an empty tree. Progressively introduce nodes, each informed by the training dataset, continuing until the dataset is completely classified or alternative termination criteria, such as maximum tree depth, are met.

Constructing a Decision Tree

  • Initial Node Construction:
    • To establish the root node, evaluate all available D features.
      • For each feature, assess various threshold values derived from the observed data within the training set.

Constructing a Decision Tree

  • For a numerical feature, the algorithm considers all possible split points (thresholds) in the feature’s range.
  • These split points are typically the midpoints between two consecutive, sorted unique values of the feature.

How Mixed Is a Node?

  • A pure node contains examples from a single class.
  • A mixed node contains examples from several classes.
  • A useful split produces children that are less mixed than their parent.

For node i, let p_{i,k} be the proportion of examples from class k.

Entropy

The entropy of node i is

H_i = -\sum_{k=1}^{K} p_{i,k}\log_2 p_{i,k}.

  • H_i=0 when the node is pure.
  • H_i increases as the classes become more evenly represented.
  • Therefore, a smaller value is better.

Binary Entropy

If p is the proportion of Gentoo penguins, then

H(p)=-p\log_2 p-(1-p)\log_2(1-p).

Code 
import matplotlib.pyplot as plt
import numpy as np

p = np.linspace(0, 1, 201)
h = np.zeros_like(p)
inside = (p > 0) & (p < 1)
h[inside] = -(
    p[inside] * np.log2(p[inside])
    + (1 - p[inside]) * np.log2(1 - p[inside])
)

fig, ax = plt.subplots(
    figsize=(4, 4),
    constrained_layout=True,
)
ax.plot(p, h, linewidth=2)
ax.scatter([0, 0.5, 1], [0, 1, 0], zorder=3)
ax.set_box_aspect(1)
ax.set(xlabel="Proportion of Gentoo penguins, p",
       ylabel="Entropy H(p) (bits)", ylim=(-0.05, 1.05))
ax.grid(alpha=0.25)
plt.show()

Entropy in Python

import numpy as np

def class_probabilities(y, classes):
    """Return the fraction of examples belonging to each class."""
    return np.array([np.mean(y == label) for label in classes])

def entropy(y, classes):
    """Measure how mixed the classes are; zero means a pure node."""
    probabilities = class_probabilities(y, classes)
    probabilities = probabilities[probabilities > 0]
    if len(probabilities) == 1:
        return 0.0
    return float(-np.sum(probabilities * np.log2(probabilities)))

Evaluating a Split

A split using feature j and threshold t produces two children:

X_{\text{left}}=\{x:x^{(j)}\leq t\}, \qquad X_{\text{right}}=\{x:x^{(j)}>t\}.

Its score is the weighted entropy:

J(j,t)= \frac{N_{\text{left}}}{N_{\text{parent}}}H_{\text{left}} + \frac{N_{\text{right}}}{N_{\text{parent}}}H_{\text{right}}.

Choose the feature and threshold that minimize J(j,t).

Three Candidate Splits

Suppose the parent contains 5 Gentoo and 5 non-Gentoo penguins.

Candidate Left child H_{\text{left}} Right child H_{\text{right}}
Messy 3 G, 2 non-G 0.971 2 G, 3 non-G 0.971
Isolate one 1 G, 0 non-G 0 4 G, 5 non-G 0.991
Useful 4 G, 1 non-G 0.722 1 G, 4 non-G 0.722

The isolated child is pure—but it contains only one example.

Why Weight the Children?

Candidate Unweighted average Weighted entropy J
Messy (0.971+0.971)/2=0.971 \frac{5}{10}(0.971)+\frac{5}{10}(0.971)=0.971
Isolate one (0+0.991)/2=\mathbf{0.496} \frac{1}{10}(0)+\frac{9}{10}(0.991)=0.892
Useful (0.722+0.722)/2=0.722 \frac{5}{10}(0.722)+\frac{5}{10}(0.722)=\mathbf{0.722}

Without weights, isolating one pure example appears best. With weights, the large mixed child receives the influence it deserves.

Evaluating a Split in Python

def weighted_entropy(y_left, y_right, classes):
    """Return the weighted entropy produced by a split."""
    n_left = len(y_left)
    n_right = len(y_right)
    n_parent = n_left + n_right

    return (
        n_left / n_parent * entropy(y_left, classes)
        + n_right / n_parent * entropy(y_right, classes)
    )

def candidate_thresholds(values):
    """Return midpoints between consecutive distinct feature values."""
    values = np.unique(values)
    return (values[:-1] + values[1:]) / 2

Greedy Split Search in Python

def find_best_split(X, y, classes, min_samples_leaf):
    best_split = None
    best_score = entropy(y, classes)
    n_candidates = 0

    for feature in range(X.shape[1]):
        for threshold in candidate_thresholds(X[:, feature]):
            go_left = X[:, feature] <= threshold
            n_left = np.sum(go_left)
            n_right = len(y) - n_left

            if min(n_left, n_right) < min_samples_leaf:
                continue

            n_candidates += 1
            score = weighted_entropy(
                y[go_left], y[~go_left], classes
            )
            if score < best_score:
                best_score = score
                best_split = (feature, float(threshold), go_left)

    return best_split, n_candidates

Complete Implementation

The complete class combines the preceding operations. Input validation and text formatting are supporting code and are not part of the central algorithm.

Read the tutorial · Download the notebook · Open in Colab

Show complete implementation 
from dataclasses import dataclass

@dataclass
class Node:
    probabilities: np.ndarray
    n_samples: int
    loss: float
    feature: int | None = None
    threshold: float | None = None
    left: "Node | None" = None
    right: "Node | None" = None

    @property
    def is_leaf(self):
        return self.feature is None


class SimpleDecisionTreeClassifier:
    """A didactic classifier with a small scikit-learn-like interface."""

    def __init__(
        self,
        max_depth=None,
        min_samples_split=2,
        min_samples_leaf=1,
    ):
        self.max_depth = max_depth
        self.min_samples_split = min_samples_split
        self.min_samples_leaf = min_samples_leaf

    def fit(self, X, y):
        X = np.asarray(X, dtype=float)
        y = np.asarray(y)
        self._validate_training_data(X, y)

        self.classes_ = np.unique(y)
        self.n_features_in_ = X.shape[1]
        self.n_candidate_splits_ = 0
        self.tree_ = self._grow_tree(X, y, depth=0)
        return self

    def _grow_tree(self, X, y, depth):
        probabilities = class_probabilities(y, self.classes_)
        node = Node(
            probabilities=probabilities,
            n_samples=len(y),
            loss=entropy(y, self.classes_),
        )

        depth_limit = self.max_depth is not None and depth >= self.max_depth
        pure_node = np.count_nonzero(probabilities) == 1
        too_small = len(y) < self.min_samples_split

        if pure_node or depth_limit or too_small:
            return node

        split, n_candidates = find_best_split(
            X, y, self.classes_, self.min_samples_leaf
        )
        self.n_candidate_splits_ += n_candidates
        if split is None:
            return node

        node.feature, node.threshold, go_left = split
        node.left = self._grow_tree(X[go_left], y[go_left], depth + 1)
        node.right = self._grow_tree(X[~go_left], y[~go_left], depth + 1)
        return node

    def _find_leaf(self, x):
        node = self.tree_
        while not node.is_leaf:
            if x[node.feature] <= node.threshold:
                node = node.left
            else:
                node = node.right
        return node

    def predict_proba(self, X):
        X = self._validate_prediction_data(X)
        return np.vstack([self._find_leaf(x).probabilities for x in X])

    def predict(self, X):
        probabilities = self.predict_proba(X)
        return self.classes_[np.argmax(probabilities, axis=1)]

    def score(self, X, y):
        return float(np.mean(self.predict(X) == np.asarray(y)))

    def export_text(self, feature_names=None):
        if feature_names is None:
            feature_names = [f"x[{j}]" for j in range(self.n_features_in_)]
        if len(feature_names) != self.n_features_in_:
            raise ValueError("feature_names must match the number of features")

        lines = []

        def visit(node, indent):
            if node.is_leaf:
                prediction = self.classes_[np.argmax(node.probabilities)]
                if isinstance(prediction, np.generic):
                    prediction = prediction.item()
                probabilities = np.round(node.probabilities, 3)
                lines.append(
                    f"{indent}predict {prediction!r} "
                    f"(p={probabilities}, n={node.n_samples})"
                )
                return

            name = feature_names[node.feature]
            lines.append(f"{indent}if {name} <= {node.threshold:.3f}:")
            visit(node.left, indent + "    ")
            lines.append(f"{indent}else:")
            visit(node.right, indent + "    ")

        visit(self.tree_, "")
        return "\n".join(lines)

    def _validate_training_data(self, X, y):
        if X.ndim != 2 or y.ndim != 1 or len(X) != len(y):
            raise ValueError("X must be 2-D and y must be 1-D with matching rows")
        if len(y) == 0:
            raise ValueError("the training set cannot be empty")
        if not np.isfinite(X).all():
            raise ValueError("missing and non-finite feature values are unsupported")
        if self.max_depth is not None and self.max_depth < 0:
            raise ValueError("max_depth must be non-negative or None")
        if self.min_samples_split < 2:
            raise ValueError("min_samples_split must be at least 2")
        if self.min_samples_leaf < 1:
            raise ValueError("min_samples_leaf must be at least 1")

    def _validate_prediction_data(self, X):
        X = np.asarray(X, dtype=float)
        if X.ndim == 1:
            X = X.reshape(1, -1)
        if X.ndim != 2 or X.shape[1] != self.n_features_in_:
            raise ValueError("X has the wrong number of features")
        if not np.isfinite(X).all():
            raise ValueError("missing and non-finite feature values are unsupported")
        return X

Training on Palmer Penguins

from palmerpenguins import load_penguins
from sklearn.model_selection import train_test_split

feature_names = ["bill_depth_mm", "body_mass_g"]
penguins_tree = load_penguins()
penguins_tree = penguins_tree[
    feature_names + ["species"]
].dropna().copy()

X = penguins_tree[feature_names].to_numpy()
y = np.where(
    penguins_tree["species"].to_numpy() == "Gentoo",
    "Gentoo",
    "Not Gentoo",
)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)

model = SimpleDecisionTreeClassifier(max_depth=2, min_samples_leaf=5)
model.fit(X_train, y_train)
<__main__.SimpleDecisionTreeClassifier at 0x119d75c40>

What Did the Tree Learn?

print(f"Test accuracy: {model.score(X_test, y_test):.3f}")
print(f"Candidate splits evaluated: {model.n_candidate_splits_}")
print(f"Class order: {model.classes_}")
print("\nLearned rules:\n")
print(model.export_text(feature_names))
Test accuracy: 1.000
Candidate splits evaluated: 312
Class order: ['Gentoo' 'Not Gentoo']

Learned rules:

if bill_depth_mm <= 16.450:
    if body_mass_g <= 3750.000:
        predict 'Not Gentoo' (p=[0. 1.], n=5)
    else:
        predict 'Gentoo' (p=[1. 0.], n=92)
else:
    if body_mass_g <= 5100.000:
        predict 'Not Gentoo' (p=[0. 1.], n=170)
    else:
        predict 'Gentoo' (p=[1. 0.], n=6)

Decision Regions

Show plotting code 
def plot_tree_boundary(X, y, model, feature_names):
    x0 = np.linspace(X[:, 0].min() - 1, X[:, 0].max() + 1, 300)
    x1 = np.linspace(X[:, 1].min() - 200, X[:, 1].max() + 200, 300)
    xx0, xx1 = np.meshgrid(x0, x1)

    grid = np.column_stack([xx0.ravel(), xx1.ravel()])
    regions = np.argmax(
        model.predict_proba(grid), axis=1
    ).reshape(xx0.shape)

    plt.figure(figsize=(9, 5))
    plt.contourf(xx0, xx1, regions, alpha=0.25, cmap="Set2")
    for label in model.classes_:
        selected = y == label
        plt.scatter(
            X[selected, 0], X[selected, 1],
            edgecolor="black", label=label
        )
    plt.xlabel(feature_names[0])
    plt.ylabel(feature_names[1])
    plt.title("Decision Tree Classification Regions")
    plt.legend()
    plt.tight_layout()
    plt.show()

plot_tree_boundary(X_train, y_train, model, feature_names)

Complete Example

Stopping Criteria

  • All the examples in a given node belong to the same class.
  • Depth of the tree would exceed max_depth.
  • Number of examples in the node is less than min_samples_split.
  • None of the splits decreases impurity sufficiently (min_impurity_decrease).
  • See documentation for other criteria.

Limitations

  • Possibly creates large trees
    • Challenge for interpretation
    • Overfitting
  • Greedy algorithm, no guarantee to find the optimal tree. (Hyafil and Rivest 1976)
  • Small changes to the data set produce vastly different trees

Large Trees

Small Changes to the Dataset

Code 
from sklearn import tree
from sklearn.metrics import classification_report, accuracy_score

# Loading the dataset

X, y = load_penguins(return_X_y = True)

target_names = ['Adelie','Chinstrap','Gentoo']

# Split the dataset into training and testing sets

for seed in (4, 7, 90, 96, 99, 2):

  print(f'Seed: {seed}')

  # Create new training and test sets based on a different random seed

  X_train, X_test, y_train, y_test = train_test_split(
      X, y, test_size=0.2, random_state=seed, stratify=y
  )

  # Creating a new classifier

  clf = tree.DecisionTreeClassifier(
      criterion="entropy", random_state=seed
  )

  # Training

  clf.fit(X_train, y_train)

  # Make predictions

  y_pred = clf.predict(X_test)

  # Plotting the tree

  tree.plot_tree(clf, 
               feature_names = X.columns,
               class_names = target_names,
               filled = True)
  plt.show()

  # Evaluating the model

  accuracy = accuracy_score(y_test, y_pred)

  report = classification_report(y_test, y_pred, target_names=target_names)

  print(f'Accuracy: {accuracy:.2f}')
  print('Classification Report:')
  print(report)

Small Changes to the Dataset

Seed: 4

Accuracy: 0.96
Classification Report:
              precision    recall  f1-score   support

      Adelie       1.00      0.90      0.95        30
   Chinstrap       0.88      1.00      0.93        14
      Gentoo       0.96      1.00      0.98        25

    accuracy                           0.96        69
   macro avg       0.95      0.97      0.95        69
weighted avg       0.96      0.96      0.96        69

Seed: 7

Accuracy: 0.94
Classification Report:
              precision    recall  f1-score   support

      Adelie       0.93      0.93      0.93        30
   Chinstrap       0.88      1.00      0.93        14
      Gentoo       1.00      0.92      0.96        25

    accuracy                           0.94        69
   macro avg       0.94      0.95      0.94        69
weighted avg       0.95      0.94      0.94        69

Seed: 90

Accuracy: 0.99
Classification Report:
              precision    recall  f1-score   support

      Adelie       0.97      1.00      0.98        30
   Chinstrap       1.00      1.00      1.00        14
      Gentoo       1.00      0.96      0.98        25

    accuracy                           0.99        69
   macro avg       0.99      0.99      0.99        69
weighted avg       0.99      0.99      0.99        69

Seed: 96

Accuracy: 0.96
Classification Report:
              precision    recall  f1-score   support

      Adelie       0.97      0.93      0.95        30
   Chinstrap       0.88      1.00      0.93        14
      Gentoo       1.00      0.96      0.98        25

    accuracy                           0.96        69
   macro avg       0.95      0.96      0.95        69
weighted avg       0.96      0.96      0.96        69

Seed: 99

Accuracy: 0.93
Classification Report:
              precision    recall  f1-score   support

      Adelie       0.96      0.90      0.93        30
   Chinstrap       0.81      0.93      0.87        14
      Gentoo       0.96      0.96      0.96        25

    accuracy                           0.93        69
   macro avg       0.91      0.93      0.92        69
weighted avg       0.93      0.93      0.93        69

Seed: 2

Accuracy: 0.97
Classification Report:
              precision    recall  f1-score   support

      Adelie       1.00      0.93      0.97        30
   Chinstrap       0.88      1.00      0.93        14
      Gentoo       1.00      1.00      1.00        25

    accuracy                           0.97        69
   macro avg       0.96      0.98      0.97        69
weighted avg       0.97      0.97      0.97        69

KNN

k-nearest neighbours (KNN)

  • Instance-based: the training examples are the model.
  • Non-parametric: the model does not have a fixed number of learned parameters.
  • Prediction assumes that nearby examples tend to have similar targets.

Learning: Store the Data

def fit(self, X, y):
    self.X_train_ = X.copy()
    self.y_train_ = y.copy()
    return self

Finding the Nearest Neighbours

For a query x and training example x_i, Euclidean distance is

d(x,x_i)=\sqrt{\sum_{j=1}^{D}\left(x^{(j)}-x_i^{(j)}\right)^2}. 

import numpy as np

def nearest_neighbors(X_train, x, n_neighbors):
    distances = np.sqrt(np.sum((X_train - x) ** 2, axis=1))
    indices = np.argsort(distances, kind="stable")[:n_neighbors]
    return indices, distances[indices]

Voting and Averaging

After selecting the k nearest examples:

  • Classification: each neighbour votes for its class.
  • Regression: average the neighbours’ numerical targets.
  • Uniform weights: every neighbour has the same influence.
  • Distance weights: closer neighbours receive weight w_i=1/d_i.

For distance-weighted regression,

\hat y(x)=\frac{\sum_{i=1}^{k}w_i y_i}{\sum_{i=1}^{k}w_i}.

Voting in Python

def voting_weights(distances, mode):
    if mode == "uniform":
        return np.ones(len(distances))

    exact_matches = distances == 0
    if np.any(exact_matches):
        return exact_matches.astype(float)

    return 1 / distances

def vote_probabilities(labels, weights, classes):
    scores = np.array([
        np.sum(weights[labels == label])
        for label in classes
    ])
    return scores / np.sum(scores)

Complete Implementation

Read the tutorial · Download the notebook · Open in Colab

Show complete implementation 
class SimpleKNeighborsClassifier:
    """A didactic classifier with a small scikit-learn-like interface."""

    def __init__(self, n_neighbors=5, weights="uniform"):
        self.n_neighbors = n_neighbors
        self.weights = weights

    def fit(self, X, y):
        X = np.asarray(X, dtype=float)
        y = np.asarray(y)
        self._validate_training_data(X, y)

        self.X_train_ = X.copy()
        self.y_train_ = y.copy()
        self.classes_ = np.unique(y)
        self.n_features_in_ = X.shape[1]
        return self

    def _predict_proba_one(self, x):
        indices, distances = nearest_neighbors(
            self.X_train_, x, self.n_neighbors
        )
        labels = self.y_train_[indices]
        weights = voting_weights(distances, self.weights)
        return vote_probabilities(labels, weights, self.classes_)

    def predict_proba(self, X):
        X = self._validate_prediction_data(X)
        return np.vstack([self._predict_proba_one(x) for x in X])

    def predict(self, X):
        probabilities = self.predict_proba(X)
        return self.classes_[np.argmax(probabilities, axis=1)]

    def score(self, X, y):
        return float(np.mean(self.predict(X) == np.asarray(y)))

    def _validate_training_data(self, X, y):
        if X.ndim != 2 or y.ndim != 1 or len(X) != len(y):
            raise ValueError(
                "X must be 2-D and y must be 1-D with matching rows"
            )
        if len(y) == 0:
            raise ValueError("the training set cannot be empty")
        if not np.isfinite(X).all():
            raise ValueError(
                "missing and non-finite feature values are unsupported"
            )
        if not isinstance(self.n_neighbors, (int, np.integer)):
            raise ValueError("n_neighbors must be an integer")
        if not 1 <= self.n_neighbors <= len(y):
            raise ValueError("n_neighbors must be between 1 and len(y)")
        if self.weights not in {"uniform", "distance"}:
            raise ValueError("weights must be 'uniform' or 'distance'")

    def _validate_prediction_data(self, X):
        if not hasattr(self, "X_train_"):
            raise ValueError("call fit before making predictions")
        X = np.asarray(X, dtype=float)
        if X.ndim == 1:
            X = X.reshape(1, -1)
        if X.ndim != 2 or X.shape[1] != self.n_features_in_:
            raise ValueError("X has the wrong number of features")
        if not np.isfinite(X).all():
            raise ValueError(
                "missing and non-finite feature values are unsupported"
            )
        return X

KNN on Palmer Penguins

from palmerpenguins import load_penguins
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler

knn_features = ["bill_length_mm", "bill_depth_mm"]
penguins_knn = load_penguins()
penguins_knn = penguins_knn[
    knn_features + ["species"]
].dropna().copy()

X_knn = penguins_knn[knn_features].to_numpy()
y_knn = penguins_knn["species"].to_numpy()
X_knn_train, X_knn_test, y_knn_train, y_knn_test = train_test_split(
    X_knn, y_knn, test_size=0.2, random_state=42, stratify=y_knn
)

knn_scaler = StandardScaler()
X_knn_train_scaled = knn_scaler.fit_transform(X_knn_train)
X_knn_test_scaled = knn_scaler.transform(X_knn_test)

for mode in ("uniform", "distance"):
    knn = SimpleKNeighborsClassifier(n_neighbors=5, weights=mode)
    knn.fit(X_knn_train_scaled, y_knn_train)
    print(f"weights='{mode}': {knn.score(X_knn_test_scaled, y_knn_test):.3f}")
weights='uniform': 0.957
weights='distance': 0.957

KNN Decision Boundaries

Show scikit-learn plotting code 
import matplotlib.pyplot as plt
from sklearn.inspection import DecisionBoundaryDisplay
from sklearn.neighbors import KNeighborsClassifier
from sklearn.pipeline import make_pipeline

configurations = [
    (1, "uniform"),
    (5, "uniform"),
    (25, "uniform"),
    (5, "distance"),
]
classes, y_encoded = np.unique(y_knn_train, return_inverse=True)

fig, axes = plt.subplots(2, 2, figsize=(11, 7), sharex=True, sharey=True)
for ax, (n_neighbors, weights) in zip(axes.ravel(), configurations):
    sklearn_knn = make_pipeline(
        StandardScaler(),
        KNeighborsClassifier(n_neighbors=n_neighbors, weights=weights),
    )
    sklearn_knn.fit(X_knn_train, y_encoded)
    DecisionBoundaryDisplay.from_estimator(
        sklearn_knn,
        X_knn_train,
        response_method="predict",
        multiclass_colors="Set2",
        alpha=0.25,
        ax=ax,
    )

    for label, name in enumerate(classes):
        selected = y_encoded == label
        ax.scatter(
            X_knn_train[selected, 0],
            X_knn_train[selected, 1],
            color=plt.get_cmap("Set2")(label),
            edgecolor="black",
            s=20,
            label=name,
        )

    ax.set_title(f"k={n_neighbors}, weights='{weights}'")
    ax.set_xlabel("Bill length (mm)")
    ax.set_ylabel("Bill depth (mm)")

handles, labels = axes[0, 0].get_legend_handles_labels()
fig.legend(handles, labels, loc="upper center", ncols=len(classes))
fig.tight_layout(rect=(0, 0, 1, 0.93))
plt.show()

KNN Limitations

  • Prediction retains and searches the training data.
  • Distances are sensitive to feature scaling and the chosen metric.
  • Neighbourhoods become less informative in high-dimensional spaces.
  • Small k can be sensitive to noise; large k can hide local structure.
  • Class imbalance can dominate a neighbourhood’s vote.

KNN Exercises and Resources

Experiment with changes in k, voting weights, distance, and feature scaling:

Summary

  • The lecture surveyed decision trees and k-nearest neighbours (KNN), and used logistic regression to illustrate a linear decision boundary.
  • We constructed a decision tree that greedily minimizes weighted entropy and a KNN classifier that stores, searches, and votes.
  • The Palmer Penguins examples illustrated how different learning algorithms produce different decision boundaries.

Prologue

Resources

References

Geurts, Pierre, Alexandre Irrthum, and Louis Wehenkel. 2009. “Supervised Learning with Decision Tree-Based Methods in Computational and Systems Biology.” Molecular bioSystems 5 (12): 1593–605. https://doi.org/10.1039/b907946g.
Hyafil, Laurent, and Ronald L. Rivest. 1976. “Constructing Optimal Binary Decision Trees Is NP-Complete.” Inf. Process. Lett. 5 (1): 15–17. https://doi.org/10.1016/0020-0190(76)90095-8.
Russell, Stuart, and Peter Norvig. 2020. Artificial Intelligence: A Modern Approach. 4th ed. Pearson. http://aima.cs.berkeley.edu/.
Stiglic, Gregor, Simon Kocbek, Igor Pernek, and Peter Kokol. 2012. “Comprehensive Decision Tree Models in Bioinformatics.” PLoS ONE 7 (3): e33812. https://doi.org/10.1371/journal.pone.0033812.

Next lecture

  • Training a linear model

Marcel Turcotte

[email protected]

School of Electrical Engineering and Computer Science (EECS)

University of Ottawa