Introduction to Artificial Neural Networks

CSI 4106 - Fall 2026

Marcel Turcotte

Version: Aug 10, 2026 11:53

Preamble

Message of the Day

I appreciate the educational videos produced by WelchLabs, particularly their guest feature on the 3Blue1Brown channel, which elucidates the mechanism by which diffusion models convert text into images. This video assumes foundational knowledge of embeddings and latent space, thus catering to an audience with a background in these concepts.

• But how do AI images and videos actually work? | Guest video by Welch Labs

These Numbers Can Make AI Dangerous [Subliminal Learning] by WelchLabs, 2025-09-04.

Message of the Day (2024)

Last year, the timing for this lecture could not have been better since John J. Hopfield and Geoffrey E. Hinton were awarded the 2024 Nobel Prize in Physics “for their fundamental discoveries and inventions enabling machine learning with artificial neural networks.”

Learning objectives

• Explain perceptrons and MLPs: structure, function, history, and limitations.
• Describe activation functions: their role in enabling complex pattern learning.
• Implement a feedforward neural network with Keras on Fashion-MNIST.
• Interpret neural network training and results: visualization and evaluation metrics.
• Familiarize with deep learning frameworks: PyTorch, TensorFlow, and Keras for model building and deployment.

As stated at the beginning of this course, there are two primary schools of thought in artificial intelligence: symbolic AI and connectionism. While the symbolic approach initially dominated the field, the connectionist approach is now more prevalent. We will now focus on connectionism.

Introduction

TensorFlow Playground

TensorFlow Playground is

playground.tensorflow.org

Neural Networks (NN)

We now shift our focus to a family of machine learning models that draw inspiration from the structure and function of biological neural networks found in animals.

AKA artificial neural networks or neural nets, abbreviated as ANN or NN.

Machine Learning Problems

• Supervised Learning: Classification, Regression

• Unsupervised Learning: Autoencoders, Self-Supervised

• Reinforcement Learning: Now an Integral Component

We will begin our exploration within the framework of supervised learning.

A neuron

In the study of artificial intelligence, it is logical to derive inspiration from the most well-understood form of intelligence: the human brain. The brain is composed of a complex network of neurons, which together form biological neural networks. Although each neuron exhibits relatively simple behavior, it is connected to thousands of other neurons, contributing to the intricate functionality of these networks.

A neuron can be conceptualized as a basic computational unit, and the complexity of brain function arises from the interconnectedness of these units.

Yann LeCun and other researchers have frequently noted that artificial neural networks used in machine learning resemble biological neural networks in much the same way that an airplane’s wings resemble those of a bird.

Attribution: Jennifer Walinga, CC BY-SA 4.0

Interconnected neurons

From biology, we essentially adopt the concept of simple computational units that are interconnected to form a network, which collectively performs complex computations.

While research into understanding biological neural networks is undeniably important, the field of artificial neural networks has incorporated only a limited number of key concepts from this research.

Attribution: Molecular Mechanism of Synaptic Function from the Howard Hughes Medical Institute (HHMI). Published on YouTube on 2018-11-15.

Neural Mind

Lakoff and Narayanan (2025)

Neurons are just cells. Alone, they are incapable of thought. But combined into circuits hooked up to our bodies in just the right ways, they have allowed human beings to survive, think, communicate, and create all of culture.

The dot product between an input vector \(\mathbf{x}\) and a weight matrix \(\mathbf{W}\), combined with a bias term and passed through a nonlinear activation function (e.g., the logistic function), defines a decision hyperplane in the input space. In isolation, such a computation is extremely limited. However, when millions or billions of these simple operations are composed into layered networks, qualitatively new behavior emerges: the ability to detect complex patterns, generate coherent language, and perform limited forms of reasoning.

Connectionist

Another characteristic of biological neural networks that we adopt is the organization of neurons into layers, particularly evident in the cerebral cortex.

The term “connectionists” comes from the idea that nodes in these models are interconnected. Instead of being explicitly programmed, these models learn their behavior through training. Deep learning is a connectionist approach.

Neural networks (NNs) consist of layers of interconnected nodes (neurons), each connection having an associated weight.

Neural networks process input data through these weighted connections, and learning occurs by adjusting the weights based on errors in the training data.

Attribution: LeNail, (2019). NN-SVG: Publication-Ready Neural Network Architecture Schematics. Journal of Open Source Software, 4(33), 747, https://doi.org/10.21105/joss.00747 (GitHub)

Hierarchy of concepts

In the book “Deep Learning” (Goodfellow et al. 2016), authors Goodfellow, Bengio, and Courville define deep learning as a subset of machine learning that enables computers to “understand the world in terms of a hierarchy of concepts.”

This hierarchical approach is one of deep learning’s most significant contributions. It reduces the need for manual feature engineering and redirects the focus toward the engineering of neural network architectures.

Attribution: LeCun et al. (2015)

Basics

Computations with neurodes

where \(x_1, x_2 \in \{0,1\}\) and \(f(z)\) is an indicator function: \[ f(z)= \begin{cases}0, & z<\theta \\ 1, & z \geq \theta\end{cases} \]

To better understand the development of modern neural networks, it is instructive to examine two earlier implementations that laid the groundwork for current advancements. This brief retrospective aims to introduce fundamental concepts relevant to contemporary networks. In this analysis, we will primarily focus on the computational aspects of neural networks, reserving the discussion on learning mechanisms for further exploration.

In mathematics, \(f(z)\), as defined above, is known as an indicator function or a characteristic function.

These neurodes have one or more binary inputs, taking a value of 0 or 1, and one binary output.

They showed that such units could implement Boolean functions such as AND, OR, and NOT.

But also that networks of such units can compute any logical proposition.

McCulloch and Pitts (1943) termed artificial neurons, neurodes, for “neuron” + “node”.

Computations with neurodes

\[ f(x_1 + x_2)= \begin{cases}0, & x_1 + x_2 <\theta \\ 1, & x_1 + x_2 \geq \theta\end{cases} \]

• With \(\theta = 2\), the neurode implements an AND logic gate.

• With \(\theta = 1\), the neurode implements an OR logic gate.

Alternatively,

\[ g(x)= \begin{cases}1, & x < \theta \\ 0, & x \geq \theta\end{cases} \]

• With \(\theta = 1\), the neurode implements a NOT logic gate.

More complex logic can be constructed by multiplying the inputs by -1, which is interpreted as inhibitory. Namely, this allows building a logical NOT.

Computations with neurodes

• Digital computations can be broken down into a sequence of logical operations, enabling neurode networks to execute any computation.

• McCulloch and Pitts (1943) did not focus on learning parameter \(\theta\).

• They introduced a machine that computes any function but cannot learn.

The period roughly from 1930 to 1950 marked a transformative shift in mathematics toward the formalization of computation. Pioneering work by Gödel, Church, and Turing not only established the theoretical limits and capabilities of computation—with Gödel’s incompleteness theorems, Church’s λ‑calculus and thesis, and Turing’s model of universal machines—but also set the stage for later developments in computer science.

McCulloch and Pitts’ 1943 model of neural networks was inspired by this early mathematical framework linking computation to aspects of intelligence, prefiguring later research in artificial intelligence.

From this work, we take the idea that networks of such units perform computations. Signal propagates from one end of the network to compute a result.

Perceptron

The buzz around artificial intelligence isn’t a new trend; in fact, it was already generating excitement back in 1958, as highlighted by a quote from the New York Times.

1958, New York Times, July 8

The Navy revealed the embryo of an electronic computer today that it expects will be able to walk, talk, see, write, reproduce itself, and be conscious of its existence.

• The Machine That Changed the World - TV Series - 1992

It is observed that artificial neural networks were introduced very early, and as early as the 1960s, they were not considered promising. Additionally, biases in the datasets used for training these networks were already noticeable.

Perceptron

Threshold logic unit

In 1957, Frank Rosenblatt developed a conceptually distinct model of a neuron known as the threshold logic unit, which he published in 1958.

In this model, both the inputs and the output of the neuron are represented as real values. Notably, each input connection has an associated weight.

The left section of the neuron, denoted by the sigma symbol, represents the computation of a weighted sum of its inputs, expressed as \(\theta_1 x^{(1)} + \theta_2 x^{(2)} + \ldots + \theta_D x^{(D)} + b\).

This sum is then processed through a step function, right section of the neuron, to generate the output.

Here, \(x^T \theta\) represents the dot product of two vectors: \(x\) and \(\theta\). Here, \(x^T\) denotes the transpose of the vector \(x\), converting it from a row vector to a column vector, allowing the dot product operation to be performed with the vector \(\theta\).

The dot product \(x^T \theta\) is then a scalar given by:

\[ x^T \theta = x^{(1)} \theta_1 + x^{(2)} \theta_2 + \cdots + x^{(D)} \theta_D \]

where \(x^{(j)}\) and \(theta_j\) are the components of the vectors \(x\) and \(\theta\), respectively.

Rosenblatt (1958)

Simple Step Functions

\(\text{Heaviside}(t)\) =

• 1, if \(t \geq 0\)

• 0, if \(t < 0\)

\(\text{sign}(t)\) =

• 1, if \(t > 0\)

• 0, if \(t = 0\)

• -1, if \(t < 0\)

Common step functions include the Heaviside step function (0 if the input is negative and 1 otherwise) or the sign function (-1 if the input is negative, 0 if the input is zero, 1 otherwise).

Notation

Add an extra feature with a fixed value of 1 to the input. Associate it with weight \(b = \theta_{0}\), where \(b\) is the bias/intercept term.

Notation

The threshold logic unit is analogous to logistic regression, with the primary distinction being the substitution of the logistic (sigmoid) function with a step function. Similar to logistic regression, the perceptron is employed for classification tasks.

\(\theta_{0} = b\) is the bias/intercept term.

Perceptron

Since the threshold logic units in this single layer also generate the output, it is referred to as the output layer.

A perceptron consists of one or more threshold logic units arranged in a single layer, with each unit connected to all inputs. This configuration is referred to as fully connected or dense.

Perceptron

Classification tasks, can be further divided into multilabel and multiclass classification.

1. Multiclass Classification:

   • In multiclass classification, each instance is assigned to one and only one class out of a set of three or more possible classes. The classes are mutually exclusive, meaning that an instance cannot belong to more than one class at the same time.

   • Example: Classifying an image as either a cat, dog, or bird. Each image can only belong to one of these categories.

2. Multilabel Classification:

   • In multilabel classification, each instance can be associated with multiple classes simultaneously. The classes are not mutually exclusive, allowing for the possibility that an instance can belong to several classes at once.

   • Example: Tagging an image with multiple attributes such as “outdoor,” “sunset,” and “beach.” The image can simultaneously belong to all these labels.

The key difference lies in the relationship between classes: multiclass classification deals with a single label per instance, while multilabel classification handles multiple labels for each instance.

As this perceptron generates multiple outputs simultaneously, it performs multiple binary predictions, making it as a multilabel classifier (can also be used as multiclass classifier).

Notation

As before, introduce an additional feature with a value of 1 to the input. Assign a bias \(b\) to each neuron. Each incoming connection implicitly has an associated weight.

Notation

• \(X\) is the input data matrix where each row corresponds to an example and each column represents one of the \(D\) features.

• \(W\) is the weight matrix, structured with one row per input (feature) and one column per neuron.

• Bias terms can be represented separately; both approaches appear in the literature. Here, \(b\) is a vector with a length equal to the number of neurons.

With neural networks, the parameters of the model are often reffered to as \(w\) (vector) or \(W\) (matrix), rather than \(\theta\).

Discussion

• The algorithm to train the perceptron closely resembles stochastic gradient descent.

  • In the interest of time and to avoid confusion, we will skip this algorithm and focus on multilayer perceptron (MLP) and its training algorithm, backpropagation.

Historical Note and Justification

This limitation also applies to other linear classifiers, such as logistic regression.

Consequently, due to these limitations and a lack of practical applications, some researchers abandoned the perceptron.

Minsky and Papert (1969) demonstrated the limitations of perceptrons, notably their inability to solve exclusive OR (XOR) classification problems: \({([0,1],\mathrm{true}), ([1,0],\mathrm{true}), ([0,0],\mathrm{false}), ([1,1],\mathrm{false})}\).

Multilayer Perceptron

A multilayer perceptron (MLP) includes an input layer and one or more layers of threshold logic units. Layers that are neither input nor output are termed hidden layers.

XOR Classification problem

\(x^{(1)}\)	\(x^{(2)}\)	\(y\)	\(o_1\)	\(o_2\)	\(o_3\)
1	0	1	0	1	1
0	1	1	0	1	1
0	0	0	0	0	0
1	1	0	1	1	0

I developed an Excel spreadsheet to verify that the proposed multilayer perceptron effectively solves the XOR classification problem.

The step function used in the above model is the Heaviside step function.

\(x^{(1)}\) and \(x^{(2)}\) are two attributes, \(y\) is the target, \(o_1\), \(o_2\), and \(o_3 = h_\theta(x)\), are the output of the top left, bottom left, and right threshold units. Clearly \(h_\theta(x) = y, \forall x \in X\). The challenge during Rosenblatt’s time was the lack of algorithms to train multi-layer networks.

Feedforward Neural Network (FNN)

The network consists of three layers: input, hidden, and output. The input layer contains two nodes, the hidden layer comprises three nodes, and the output layer has two nodes. Additional hidden layers and nodes per layer can be added, which will be discussed later.

It is often useful to include explicit input nodes that do not perform calculations, known as input units or input neurons. These nodes act as placeholders to introduce input features into the network, passing data directly to the next layer without transformation. In the network diagram, these are the light blue nodes on the left, labeled 1 and 2. Typically, the number of input units corresponds to the number of features.

For clarity, nodes are labeled to facilitate discussion of the weights between them, such as \(w_{1,5}\) between nodes 1 and 5. Similarly, the output of a node is denoted by \(o_k\), where \(k\) represents the node’s label. For example, for \(k=3\), the output would be \(o_3\).

Information in this architecture flows unidirectionally—from left to right, moving from input to output. Consequently, it is termed a feedforward neural network.

Forward Pass (Computatation)

\(o3 = \sigma(w_{13} x^{(1)}+ w_{23} x^{(2)} + b_3)\)

\(o4 = \sigma(w_{14} x^{(1)}+ w_{24} x^{(2)} + b_4)\)

\(o5 = \sigma(w_{15} x^{(1)}+ w_{25} x^{(2)} + b_5)\)

\(o6 = \sigma(w_{36} o_3 + w_{46} o_4 + w_{56} o_5 + b_6)\)

\(o7 = \sigma(w_{37} o_3 + w_{47} o_4 + w_{57} o_5 + b_7)\)

To simplify the figure, I have opted not to display the bias terms, though they remain crucial components. Specifically, \(b_3\) represents the bias term associated with node 3.

If bias terms were not significant, the training process would naturally reduce them to zero. Bias terms are essential as they enable the adjustment of the decision boundary, allowing the model to learn more complex patterns that weights alone cannot capture. By offering additional degrees of freedom, they also contribute to faster convergence during training.

First, it’s important to understand the information flow: this network computes two outputs from its inputs.

Forward Pass (Computatation)

import numpy as np

# Sigmoid function

def sigma(x):
    return 1 / (1 + np.exp(-x))

# Input (two attributes) vector, one example of our trainig set

x1, x2 = (0.5, 0.9)

# Initializing the weights of layers 2 and 3 to random values

w13, w14, w15, w23, w24, w25 = np.random.uniform(low=-1, high=1, size=6)
w36, w46, w56, w37, w47, w57 = np.random.uniform(low=-1, high=1, size=6)

# Initializing all 5 bias terms to random values

b3, b4, b5, b6, b7 = np.random.uniform(low=-1, high=1, size=5)

o3 = sigma(w13 * x1 + w23 * x2 + b3)
o4 = sigma(w14 * x1 + w24 * x2 + b4)
o5 = sigma(w15 * x1 + w25 * x2 + b5)
o6 = sigma(w36 * o3 + w46 * o4 + w56 * o5 + b6)
o7 = sigma(w37 * o3 + w47 * o4 + w57 * o5 + b7)

(o6, o7)

(np.float64(0.578103468529883), np.float64(0.5578610451211033))

The example above illustrates the computation process with specific values. Before training a neural network, it is standard practice to initialize the weights and biases with random values. Gradient descent is then employed to iteratively adjust these parameters, aiming to minimize the loss function.

Forward Pass (Computatation)

The information flow remains consistent even in more complex networks. Networks with many layers are called deep neural networks (DNN).

Produced using NN-SVG, LeNail (2019).

Forward Pass (Computatation)

Same network with bias terms shown.

Produced using NN-SVG, LeNail (2019).

Activation Function

• As will be discussed later, the training algorithm, known as backpropagation, employs gradient descent, necessitating the calculation of the partial derivatives of the loss function.

• The step function in the multilayer perceptron had to be replaced, as it consists only of flat surfaces. Gradient descent cannot progress on flat surfaces due to their zero derivative.

Activation Function

• Nonlinear activation functions are paramount because, without them, multiple layers in the network would only compute a linear function of the inputs.

• According to the Universal Approximation Theorem, sufficiently large deep networks with nonlinear activation functions can approximate any continuous function. See Universal Approximation Theorem.

Sigmoid

import matplotlib.pyplot as plt

# Sigmoid function
def sigmoid(x):
    return 1 / (1 + np.exp(-x))

# Generate x values
x = np.linspace(-10, 10, 400)

# Compute y values for the sigmoid function
y = sigmoid(x)

# Create a figure and remove axes and grid
fig, ax = plt.subplots()
ax.plot(x, y, color='black', linewidth=2)  # Keep the curve opaque

plt.grid(True)

# Set transparent background for the figure and axes
fig.patch.set_alpha(0)  # Transparent background for the figure

# Save or display the plot with transparent background
# plt.savefig('sigmoid_plot.png', transparent=True, bbox_inches='tight', pad_inches=0)
plt.show()

\[ \sigma(t) = \frac{1}{1 + e^{-t}} \]

Hyperbolic Tangent Function

# Generate x values
x = np.linspace(-10, 10, 400)

# Compute y values for the sigmoid function
y = np.tanh(x)

# Create a figure and remove axes and grid
fig, ax = plt.subplots()
ax.plot(x, y, color='black', linewidth=2)  # Keep the curve opaque

plt.grid(True)

# Set transparent background for the figure and axes
fig.patch.set_alpha(0)  # Transparent background for the figure

# Save or display the plot with transparent background
# plt.savefig('sigmoid_plot.png', transparent=True, bbox_inches='tight', pad_inches=0)
plt.show()

\[ \tanh(t) = 2 \sigma(2t) - 1 \]

This S-shaped curve, similar to the sigmoid function, produces output values ranging from -1 to 1. According to Géron (2022), this range helps each layer’s output to be approximately centered around 0 at the start of training, thereby accelerating convergence.

Rectified linear unit function (ReLU)

# Generate x values
x = np.linspace(-10, 10, 400)

# Compute y values for the sigmoid function
y = np.maximum(0, x)

# Create a figure and remove axes and grid
fig, ax = plt.subplots()
ax.plot(x, y, color='black', linewidth=2)  # Keep the curve opaque

plt.grid(True)

# Set transparent background for the figure and axes
fig.patch.set_alpha(0)  # Transparent background for the figure

# Save or display the plot with transparent background
# plt.savefig('sigmoid_plot.png', transparent=True, bbox_inches='tight', pad_inches=0)
plt.show()

\[ \mathrm{ReLU}(t) = \max(0, t) \]

Although the ReLU function is not differentiable at \(t=0\) and has a derivative of 0 for \(t<0\), it performs quite well in practice and is computationally efficient. Consequently, it has become the default activation function.

Common Activation Functions

from scipy.special import expit as sigmoid

def relu(z):
    return np.maximum(0, z)

def derivative(f, z, eps=0.000001):
    return (f(z + eps) - f(z - eps))/(2 * eps)

max_z = 4.5
z = np.linspace(-max_z, max_z, 200)

plt.figure(figsize=(11, 3.1))

plt.subplot(121)
plt.plot([-max_z, 0], [0, 0], "r-", linewidth=2, label="Heaviside")
plt.plot(z, relu(z), "m-.", linewidth=2, label="ReLU")
plt.plot([0, 0], [0, 1], "r-", linewidth=0.5)
plt.plot([0, max_z], [1, 1], "r-", linewidth=2)
plt.plot(z, sigmoid(z), "g--", linewidth=2, label="Sigmoid")
plt.plot(z, np.tanh(z), "b-", linewidth=1, label="Tanh")
plt.grid(True)
plt.title("Activation functions")
plt.axis([-max_z, max_z, -1.65, 2.4])
plt.gca().set_yticks([-1, 0, 1, 2])
plt.legend(loc="lower right", fontsize=13)

plt.subplot(122)
plt.plot(z, derivative(np.sign, z), "r-", linewidth=2, label="Heaviside")
plt.plot(0, 0, "ro", markersize=5)
plt.plot(0, 0, "rx", markersize=10)
plt.plot(z, derivative(sigmoid, z), "g--", linewidth=2, label="Sigmoid")
plt.plot(z, derivative(np.tanh, z), "b-", linewidth=1, label="Tanh")
plt.plot([-max_z, 0], [0, 0], "m-.", linewidth=2)
plt.plot([0, max_z], [1, 1], "m-.", linewidth=2)
plt.plot([0, 0], [0, 1], "m-.", linewidth=1.2)
plt.plot(0, 1, "mo", markersize=5)
plt.plot(0, 1, "mx", markersize=10)
plt.grid(True)
plt.title("Derivatives")
plt.axis([-max_z, max_z, -0.2, 1.2])

plt.show()

Géron (2022) – 10_neural_nets_with_keras.ipynb

Universal Approximation

Definition

The universal approximation theorem (UAT) states that a feedforward neural network with a single hidden layer containing a finite number of neurons can approximate any continuous function on a compact subset of \(\mathbb{R}^n\), given appropriate weights and activation functions.

A compact region in \(\mathbb R^n\) means a set that is both closed (it contains its boundary) and bounded (it lies within some finite ball). The requirement of compactness in the universal-approximation theorem ensures the function domain is ‘finite and well-behaved’ so the theorem’s guarantee applies.

Cybenko (1989); Hornik et al. (1989)

Single Hidden Layer

\[ y = \sum_{i=1}^N \alpha_i \sigma(w_{1,i} x + b_i) \]

Notation adapted to follow that of Cybenko (1989).

Effect of Varying w

def logistic(x, w, b):
    """Compute the logistic function with parameters w and b."""
    return 1 / (1 + np.exp(-(w * x + b)))

# Define a range for x values.
x = np.linspace(-10, 10, 400)

# Plot 1: Varying w (steepness) with b fixed at 0.
plt.figure(figsize=(6,4))
w_values = [0.5, 1, 2, 5]  # different steepness values
b = 0  # fixed bias

for w in w_values:
    plt.plot(x, logistic(x, w, b), label=f'w = {w}, b = {b}')
plt.title('Effect of Varying w (with b = 0)')
plt.xlabel('x')
plt.ylabel(r'$\sigma(wx+b)$')
plt.legend()
plt.grid(True)

plt.show()

Sigmoid activation function: \(\sigma(wx+b)\).

Effect of Varying b

# Plot 2: Varying b (horizontal shift) with w fixed at 1.
plt.figure(figsize=(6,4))
w = 1  # fixed steepness
b_values = [-5, -2, 0, 2, 5]  # different bias values

for b in b_values:
    plt.plot(x, logistic(x, w, b), label=f'w = {w}, b = {b}')
plt.title('Effect of Varying b (with w = 1)')
plt.xlabel('x')
plt.ylabel(r'$\sigma(wx+b)$')
plt.legend()
plt.grid(True)

plt.show()

Sigmoid activation function: \(\sigma(wx+b)\).

Effect of Varying w

def relu(x, w, b):
    """Compute the ReLU activation with parameters w and b."""
    return np.maximum(0, w * x + b)

# Define a range for x values.
x = np.linspace(-10, 10, 400)

# Plot 1: Varying w (scaling) with b fixed at 0.
plt.figure(figsize=(6,4))
w_values = [0.5, 1, 2, 5]  # different scaling values
b = 0  # fixed bias

for w in w_values:
    plt.plot(x, relu(x, w, b), label=f'w = {w}, b = {b}')
plt.title('Effect of Varying w (with b = 0) on ReLU Activation')
plt.xlabel('x')
plt.ylabel('ReLU(wx+b)')
plt.legend()
plt.grid(True)

plt.show()

ReLU activation function: np.maximum(0, w * x + b).

Effect of Varying b

# Plot 2: Varying b (horizontal shift) with w fixed at 1.
plt.figure(figsize=(6,4))
w = 1  # fixed scaling
b_values = [-5, -2, 0, 2, 5]  # different bias values

for b in b_values:
    plt.plot(x, relu(x, w, b), label=f'w = {w}, b = {b}')
plt.title('Effect of Varying b (with w = 1) on ReLU Activation')
plt.xlabel('x')
plt.ylabel('ReLU(wx+b)')
plt.legend()
plt.grid(True)

plt.show()

ReLU activation function: np.maximum(0, w * x + b).

Single Hidden Layer

\[ y = \sum_{i=1}^N \alpha_i \sigma(w_{1,i} x + b_i) \]

See also: Chapter 4: A visual proof that neural nets can compute any function, Neural Networks and Deep Learning by Michael Nielsen.

Demonstration with code

# Defining the function to be approximated

def f(x):
  return 2 * x**3 + 4 * x**2 - 5 * x + 1

# Generating a dataset, x in [-4,2), f(x) as above

X = 6 * np.random.rand(1000, 1) - 4

y = f(X.flatten())

Increasing the number of neurons

from sklearn.neural_network import MLPRegressor
from sklearn.model_selection import train_test_split

X_train, X_valid, y_train, y_valid = train_test_split(X, y, test_size=0.1, random_state=42)

models = []

sizes = [1, 2, 5, 10, 100]

for i, n in enumerate(sizes):

  models.append(MLPRegressor(hidden_layer_sizes=[n], max_iter=5000, random_state=42))

  models[i].fit(X_train, y_train) 

MLPRegressor is a multi-layer perceptron regressor from sklearn. Its default activation function is relu.

Increasing the number of neurons

# Create a colormap
colors = plt.colormaps['cool'].resampled(len(sizes))

X_valid = np.sort(X_valid,axis=0)

for i, n in enumerate(sizes):

  y_pred = models[i].predict(X_valid)

  plt.plot(X_valid, y_pred, "-", color=colors(i), label="Number of neurons = {}".format(n))

y_true = f(X_valid)
plt.plot(X_valid, y_true, "r.", label='Actual')

plt.legend()
plt.show()

In the example above, I retained only 10% of the data as the test set because the function being approximated is straightforward and noise-free. This decision was made to ensure that the true curve does not overshadow the other results.

Increasing the number of neurons

for i, n in enumerate(sizes):

  plt.plot(models[i].loss_curve_, "-", color=colors(i), label="Number of neurons = {}".format(n))

plt.title('MLPRegressor Loss Curves')
plt.xlabel('Iterations')
plt.ylabel('Loss')

plt.legend()
plt.show()

As expected, increasing neuron count reduces loss.

Universal Approximation

The video effectively elucidates key concepts (terminology) in neural networks, including nodes, layers, weights, and activation functions. It demonstrates the process of summing activation outputs from a preceding layer, akin to the aggregation of curves. Additionally, the video illustrates how scaling an output by a weight not only alters the amplitude of a curve but also inverts its orientation when the weight is negative. Moreover, it clearly depicts the function of bias terms in vertically shifting the curve, contingent on the sign of the bias.

This video effectively conveys the underlying intuition of the universal approximation theorem. (18m 53s)

Let’s code

Frameworks

PyTorch and TensorFlow are the leading platforms for deep learning.

• PyTorch has gained considerable traction in the research community. Initially developed by Meta AI, it is now part of the Linux Foundation.

• TensorFlow, created by Google, is widely adopted in industry for deploying models in production environments.

Official PyTorch Documentary: Powering the AI Revolution

Keras

Keras is a high-level API designed to build, train, evaluate, and execute models across various backends, including PyTorch, TensorFlow, and JAX, Google’s high-performance platform.

As highlighted in previous Quotes of the Day, François Chollet, a Google engineer, is the originator and one of the primary developers of the Keras project.

Keras is powerful enough for most projects.

Fashion-MNIST dataset

“Fashion-MNIST is a dataset of Zalando’s article images—consisting of a training set of 60,000 examples and a test set of 10,000 examples. Each example is a 28x28 grayscale image, associated with a label from 10 classes.”

Attribution: Géron (2022) – 10_neural_nets_with_keras.ipynb

Loading

import tensorflow as tf

fashion_mnist = tf.keras.datasets.fashion_mnist

(X_train_full, y_train_full), (X_test, y_test) = fashion_mnist.load_data()

X_train, y_train = X_train_full[:-5000], y_train_full[:-5000]
X_valid, y_valid = X_train_full[-5000:], y_train_full[-5000:]

Setting aside 5000 examples as a validation set.

Exploration

X_train.shape

(55000, 28, 28)

X_train.dtype

dtype('uint8')

Transforming the pixel intensities from integers in the range 0 to 255 to floats in the range 0 to 1.

X_train = X_train / 255.0
X_valid = X_valid / 255.0

What are these images anyway!

plt.figure(figsize=(2, 2))
plt.imshow(X_train[0], cmap="binary")
plt.axis('off')
plt.show()

y_train

array([9, 0, 0, ..., 9, 0, 2], shape=(55000,), dtype=uint8)

Since the labels are integers, 0 to 9. Class names will become handy.

class_names = ["T-shirt/top", "Trouser", "Pullover", "Dress", "Coat",
               "Sandal", "Shirt", "Sneaker", "Bag", "Ankle boot"]

First 40 images

n_rows = 4
n_cols = 10
plt.figure(figsize=(n_cols * 1.2, n_rows * 1.2))
for row in range(n_rows):
    for col in range(n_cols):
        index = n_cols * row + col
        plt.subplot(n_rows, n_cols, index + 1)
        plt.imshow(X_train[index], cmap="binary", interpolation="nearest")
        plt.axis('off')
        plt.title(class_names[y_train[index]])
plt.subplots_adjust(wspace=0.2, hspace=0.5)
plt.show()

First 40 images

Creating a model

tf.random.set_seed(42)

model = tf.keras.Sequential()

model.add(tf.keras.layers.InputLayer(shape=[28, 28]))
model.add(tf.keras.layers.Flatten())
model.add(tf.keras.layers.Dense(300, activation="relu"))
model.add(tf.keras.layers.Dense(100, activation="relu"))
model.add(tf.keras.layers.Dense(10, activation="softmax"))

model.summary()

model.summary()

Model: "sequential"

┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Layer (type)                    ┃ Output Shape           ┃       Param # ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ flatten (Flatten)               │ (None, 784)            │             0 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense (Dense)                   │ (None, 300)            │       235,500 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_1 (Dense)                 │ (None, 100)            │        30,100 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_2 (Dense)                 │ (None, 10)             │         1,010 │
└─────────────────────────────────┴────────────────────────┴───────────────┘

 Total params: 266,610 (1.02 MB)

 Trainable params: 266,610 (1.02 MB)

 Non-trainable params: 0 (0.00 B)

As observed, dense_3 has \(235,500\) parameters, while \(784 \times 300 = 235,200\).

Could you explain the origin of the additional parameters?

Similarly, dense_3 has \(30,100\) parameters, while \(300 \times 100 = 30,000\).

Can you explain why?

Creating a model (alternative)

# extra code – clear the session to reset the name counters
tf.keras.backend.clear_session()
tf.random.set_seed(42)

model = tf.keras.Sequential([
    tf.keras.Input(shape=(28, 28)),
    tf.keras.layers.Flatten(),
    tf.keras.layers.Dense(300, activation="relu"),
    tf.keras.layers.Dense(100, activation="relu"),
    tf.keras.layers.Dense(10, activation="softmax")
])

model.summary()

model.summary()

Model: "sequential"

┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Layer (type)                    ┃ Output Shape           ┃       Param # ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ flatten (Flatten)               │ (None, 784)            │             0 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense (Dense)                   │ (None, 300)            │       235,500 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_1 (Dense)                 │ (None, 100)            │        30,100 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_2 (Dense)                 │ (None, 10)             │         1,010 │
└─────────────────────────────────┴────────────────────────┴───────────────┘

 Total params: 266,610 (1.02 MB)

 Trainable params: 266,610 (1.02 MB)

 Non-trainable params: 0 (0.00 B)

Compiling the model

model.compile(loss="sparse_categorical_crossentropy",
              optimizer="sgd",
              metrics=["accuracy"])

sparse_categorical_crossentropy is the appropriate function for a multiclass classification problem (more later).

The method compile allows to set the loss function, as well as other parameters. Keras then prepares the model for training.

Training the model

history = model.fit(X_train, y_train, epochs=30,
                    validation_data=(X_valid, y_valid))

Epoch 1/30


   1/1719 ━━━━━━━━━━━━━━━━━━━━ 4:28 156ms/step - accuracy: 0.1250 - loss: 2.3250

  75/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 679us/step - accuracy: 0.4679 - loss: 1.8039  

 151/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 671us/step - accuracy: 0.5575 - loss: 1.5078

 228/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 664us/step - accuracy: 0.6062 - loss: 1.3254

 305/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 662us/step - accuracy: 0.6331 - loss: 1.2066

 383/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 659us/step - accuracy: 0.6536 - loss: 1.1232

 460/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 658us/step - accuracy: 0.6711 - loss: 1.0581

 536/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 659us/step - accuracy: 0.6844 - loss: 1.0094

 613/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 658us/step - accuracy: 0.6956 - loss: 0.9686

 690/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 658us/step - accuracy: 0.7064 - loss: 0.9325

 765/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 660us/step - accuracy: 0.7145 - loss: 0.9041

 837/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 663us/step - accuracy: 0.7223 - loss: 0.8792

 912/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 664us/step - accuracy: 0.7291 - loss: 0.8543

 987/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 664us/step - accuracy: 0.7354 - loss: 0.8329

1061/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 666us/step - accuracy: 0.7399 - loss: 0.8156

1137/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 665us/step - accuracy: 0.7441 - loss: 0.7993

1213/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 665us/step - accuracy: 0.7479 - loss: 0.7858

1288/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 665us/step - accuracy: 0.7521 - loss: 0.7720

1365/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 665us/step - accuracy: 0.7564 - loss: 0.7581

1443/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 664us/step - accuracy: 0.7601 - loss: 0.7466

1521/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 663us/step - accuracy: 0.7633 - loss: 0.7351

1597/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 663us/step - accuracy: 0.7658 - loss: 0.7255

1673/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 663us/step - accuracy: 0.7679 - loss: 0.7168

1719/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 761us/step - accuracy: 0.7691 - loss: 0.7115 - val_accuracy: 0.8286 - val_loss: 0.5011

Epoch 2/30


   1/1719 ━━━━━━━━━━━━━━━━━━━━ 14s 8ms/step - accuracy: 0.8438 - loss: 0.5071

  70/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 730us/step - accuracy: 0.8313 - loss: 0.5025

 142/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 717us/step - accuracy: 0.8195 - loss: 0.5305

 211/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 721us/step - accuracy: 0.8220 - loss: 0.5203

 281/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 720us/step - accuracy: 0.8212 - loss: 0.5207

 351/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 720us/step - accuracy: 0.8231 - loss: 0.5161

 424/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 715us/step - accuracy: 0.8241 - loss: 0.5124

 496/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 712us/step - accuracy: 0.8241 - loss: 0.5106

 568/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 711us/step - accuracy: 0.8254 - loss: 0.5090

 642/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 708us/step - accuracy: 0.8271 - loss: 0.5068

 714/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 707us/step - accuracy: 0.8281 - loss: 0.5052

 787/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 705us/step - accuracy: 0.8281 - loss: 0.5039

 859/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 705us/step - accuracy: 0.8285 - loss: 0.5030

 933/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 703us/step - accuracy: 0.8299 - loss: 0.4982

1006/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 702us/step - accuracy: 0.8310 - loss: 0.4952

1075/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 704us/step - accuracy: 0.8312 - loss: 0.4939

1149/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 702us/step - accuracy: 0.8316 - loss: 0.4924

1222/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 702us/step - accuracy: 0.8318 - loss: 0.4923

1295/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 701us/step - accuracy: 0.8323 - loss: 0.4902

1366/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 701us/step - accuracy: 0.8331 - loss: 0.4879

1438/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 701us/step - accuracy: 0.8337 - loss: 0.4866

1509/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 702us/step - accuracy: 0.8343 - loss: 0.4846

1581/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 701us/step - accuracy: 0.8342 - loss: 0.4838

1652/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 702us/step - accuracy: 0.8339 - loss: 0.4828

1719/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 750us/step - accuracy: 0.8337 - loss: 0.4820 - val_accuracy: 0.8404 - val_loss: 0.4506

Epoch 3/30


   1/1719 ━━━━━━━━━━━━━━━━━━━━ 12s 7ms/step - accuracy: 0.8750 - loss: 0.4523

  73/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 698us/step - accuracy: 0.8476 - loss: 0.4403

 146/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 695us/step - accuracy: 0.8384 - loss: 0.4668

 221/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 688us/step - accuracy: 0.8404 - loss: 0.4583

 294/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 688us/step - accuracy: 0.8393 - loss: 0.4603

 367/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 688us/step - accuracy: 0.8414 - loss: 0.4567

 441/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 687us/step - accuracy: 0.8415 - loss: 0.4570

 516/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 685us/step - accuracy: 0.8427 - loss: 0.4540

 590/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 684us/step - accuracy: 0.8443 - loss: 0.4523

 664/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 684us/step - accuracy: 0.8447 - loss: 0.4520

 737/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 684us/step - accuracy: 0.8455 - loss: 0.4511

 810/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 684us/step - accuracy: 0.8459 - loss: 0.4489

 887/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 682us/step - accuracy: 0.8454 - loss: 0.4497

 962/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 682us/step - accuracy: 0.8464 - loss: 0.4459

1037/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 681us/step - accuracy: 0.8464 - loss: 0.4444

1114/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 679us/step - accuracy: 0.8461 - loss: 0.4440

1184/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 682us/step - accuracy: 0.8465 - loss: 0.4425

1257/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 682us/step - accuracy: 0.8463 - loss: 0.4438

1333/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 681us/step - accuracy: 0.8470 - loss: 0.4412

1409/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 680us/step - accuracy: 0.8476 - loss: 0.4395

1483/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 680us/step - accuracy: 0.8478 - loss: 0.4393

1557/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 680us/step - accuracy: 0.8482 - loss: 0.4379

1630/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 681us/step - accuracy: 0.8479 - loss: 0.4374

1701/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 682us/step - accuracy: 0.8476 - loss: 0.4371

1719/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 729us/step - accuracy: 0.8475 - loss: 0.4371 - val_accuracy: 0.8476 - val_loss: 0.4274

Epoch 4/30


   1/1719 ━━━━━━━━━━━━━━━━━━━━ 12s 7ms/step - accuracy: 0.8438 - loss: 0.4372

  71/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 715us/step - accuracy: 0.8583 - loss: 0.4045

 145/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 697us/step - accuracy: 0.8487 - loss: 0.4342

 217/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 698us/step - accuracy: 0.8510 - loss: 0.4265

 291/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 694us/step - accuracy: 0.8489 - loss: 0.4291

 364/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 692us/step - accuracy: 0.8513 - loss: 0.4246

 436/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 693us/step - accuracy: 0.8514 - loss: 0.4249

 509/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 693us/step - accuracy: 0.8523 - loss: 0.4226

 582/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 693us/step - accuracy: 0.8533 - loss: 0.4216

 653/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 694us/step - accuracy: 0.8546 - loss: 0.4200

 720/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 700us/step - accuracy: 0.8549 - loss: 0.4198

 788/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 703us/step - accuracy: 0.8543 - loss: 0.4202

 862/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 701us/step - accuracy: 0.8539 - loss: 0.4215

 935/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 700us/step - accuracy: 0.8547 - loss: 0.4175

1008/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 699us/step - accuracy: 0.8550 - loss: 0.4158

1081/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 699us/step - accuracy: 0.8553 - loss: 0.4146

1155/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 698us/step - accuracy: 0.8552 - loss: 0.4144

1227/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 698us/step - accuracy: 0.8552 - loss: 0.4149

1303/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 696us/step - accuracy: 0.8554 - loss: 0.4133

1378/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 695us/step - accuracy: 0.8562 - loss: 0.4114

1450/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 695us/step - accuracy: 0.8562 - loss: 0.4115

1520/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 696us/step - accuracy: 0.8566 - loss: 0.4104

1593/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 696us/step - accuracy: 0.8566 - loss: 0.4104

1661/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 698us/step - accuracy: 0.8561 - loss: 0.4106

1719/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 745us/step - accuracy: 0.8560 - loss: 0.4100 - val_accuracy: 0.8506 - val_loss: 0.4154

Epoch 5/30


   1/1719 ━━━━━━━━━━━━━━━━━━━━ 12s 7ms/step - accuracy: 0.8438 - loss: 0.4306

  72/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 705us/step - accuracy: 0.8641 - loss: 0.3806

 145/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 696us/step - accuracy: 0.8554 - loss: 0.4116

 217/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 697us/step - accuracy: 0.8579 - loss: 0.4047

 289/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 697us/step - accuracy: 0.8556 - loss: 0.4075

 359/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 702us/step - accuracy: 0.8572 - loss: 0.4037

 429/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 705us/step - accuracy: 0.8584 - loss: 0.4021

 501/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 705us/step - accuracy: 0.8580 - loss: 0.4015

 573/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 705us/step - accuracy: 0.8591 - loss: 0.4003

 646/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 703us/step - accuracy: 0.8607 - loss: 0.3981

 714/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 706us/step - accuracy: 0.8611 - loss: 0.3986

 783/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 708us/step - accuracy: 0.8604 - loss: 0.3993

 856/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 707us/step - accuracy: 0.8600 - loss: 0.4005

 930/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 705us/step - accuracy: 0.8609 - loss: 0.3964

1003/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 704us/step - accuracy: 0.8610 - loss: 0.3953

1077/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 703us/step - accuracy: 0.8612 - loss: 0.3941

1150/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 702us/step - accuracy: 0.8615 - loss: 0.3931

1223/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 701us/step - accuracy: 0.8614 - loss: 0.3940

1294/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 701us/step - accuracy: 0.8615 - loss: 0.3926

1365/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 702us/step - accuracy: 0.8622 - loss: 0.3913

1436/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 702us/step - accuracy: 0.8626 - loss: 0.3910

1509/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 701us/step - accuracy: 0.8627 - loss: 0.3902

1583/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 700us/step - accuracy: 0.8626 - loss: 0.3904

1654/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 700us/step - accuracy: 0.8621 - loss: 0.3906

1719/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 745us/step - accuracy: 0.8620 - loss: 0.3902 - val_accuracy: 0.8526 - val_loss: 0.4053

Epoch 6/30


   1/1719 ━━━━━━━━━━━━━━━━━━━━ 12s 7ms/step - accuracy: 0.8125 - loss: 0.4215

  73/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 696us/step - accuracy: 0.8711 - loss: 0.3643

 136/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 742us/step - accuracy: 0.8635 - loss: 0.3934

 145/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 1ms/step - accuracy: 0.8616 - loss: 0.3934  

 190/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 1ms/step - accuracy: 0.8646 - loss: 0.3859

 259/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 1ms/step - accuracy: 0.8644 - loss: 0.3848

 328/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 1ms/step - accuracy: 0.8641 - loss: 0.3833

 394/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 976us/step - accuracy: 0.8620 - loss: 0.3872

 463/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 939us/step - accuracy: 0.8627 - loss: 0.3856

 533/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 910us/step - accuracy: 0.8632 - loss: 0.3851

 605/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 885us/step - accuracy: 0.8643 - loss: 0.3821

 675/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 868us/step - accuracy: 0.8654 - loss: 0.3816

 739/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 861us/step - accuracy: 0.8657 - loss: 0.3819

 807/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 851us/step - accuracy: 0.8657 - loss: 0.3813

 877/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 840us/step - accuracy: 0.8651 - loss: 0.3831

 950/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 829us/step - accuracy: 0.8660 - loss: 0.3800

1023/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 818us/step - accuracy: 0.8663 - loss: 0.3781

1095/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 810us/step - accuracy: 0.8668 - loss: 0.3770

1166/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 804us/step - accuracy: 0.8667 - loss: 0.3769

1240/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 797us/step - accuracy: 0.8665 - loss: 0.3777

1313/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 791us/step - accuracy: 0.8669 - loss: 0.3759

1387/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 785us/step - accuracy: 0.8672 - loss: 0.3751

1455/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 783us/step - accuracy: 0.8676 - loss: 0.3748

1527/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 779us/step - accuracy: 0.8677 - loss: 0.3743

1594/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 778us/step - accuracy: 0.8675 - loss: 0.3742

1661/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 777us/step - accuracy: 0.8669 - loss: 0.3749

1719/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 830us/step - accuracy: 0.8669 - loss: 0.3743 - val_accuracy: 0.8546 - val_loss: 0.3969

Epoch 7/30


   1/1719 ━━━━━━━━━━━━━━━━━━━━ 14s 8ms/step - accuracy: 0.8125 - loss: 0.4032

  67/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 766us/step - accuracy: 0.8759 - loss: 0.3483

 137/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 744us/step - accuracy: 0.8686 - loss: 0.3783

 206/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 740us/step - accuracy: 0.8683 - loss: 0.3736

 274/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 741us/step - accuracy: 0.8670 - loss: 0.3736

 337/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 753us/step - accuracy: 0.8686 - loss: 0.3720

 392/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 775us/step - accuracy: 0.8673 - loss: 0.3738

 454/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 781us/step - accuracy: 0.8689 - loss: 0.3715

 521/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 778us/step - accuracy: 0.8682 - loss: 0.3709

 591/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 771us/step - accuracy: 0.8692 - loss: 0.3687

 662/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 764us/step - accuracy: 0.8703 - loss: 0.3676

 731/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 761us/step - accuracy: 0.8711 - loss: 0.3675

 800/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 758us/step - accuracy: 0.8716 - loss: 0.3667

 870/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 755us/step - accuracy: 0.8708 - loss: 0.3688

 937/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 755us/step - accuracy: 0.8715 - loss: 0.3657

1004/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 754us/step - accuracy: 0.8717 - loss: 0.3647

1073/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 753us/step - accuracy: 0.8720 - loss: 0.3636

1144/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 750us/step - accuracy: 0.8724 - loss: 0.3629

1212/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 749us/step - accuracy: 0.8723 - loss: 0.3634

1281/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 748us/step - accuracy: 0.8720 - loss: 0.3630

1350/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 747us/step - accuracy: 0.8726 - loss: 0.3618

1421/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 745us/step - accuracy: 0.8731 - loss: 0.3606

1490/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 744us/step - accuracy: 0.8728 - loss: 0.3611

1554/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 746us/step - accuracy: 0.8728 - loss: 0.3605

1624/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 745us/step - accuracy: 0.8723 - loss: 0.3607

1685/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 748us/step - accuracy: 0.8720 - loss: 0.3612

1719/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 802us/step - accuracy: 0.8719 - loss: 0.3609 - val_accuracy: 0.8586 - val_loss: 0.3897

Epoch 8/30


   1/1719 ━━━━━━━━━━━━━━━━━━━━ 13s 8ms/step - accuracy: 0.8125 - loss: 0.3871

  75/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 682us/step - accuracy: 0.8763 - loss: 0.3408

 142/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 717us/step - accuracy: 0.8691 - loss: 0.3658

 207/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 736us/step - accuracy: 0.8705 - loss: 0.3600

 283/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 717us/step - accuracy: 0.8708 - loss: 0.3602

 358/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 708us/step - accuracy: 0.8729 - loss: 0.3600

 434/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 700us/step - accuracy: 0.8736 - loss: 0.3592

 510/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 695us/step - accuracy: 0.8729 - loss: 0.3581

 585/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 692us/step - accuracy: 0.8735 - loss: 0.3570

 660/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 689us/step - accuracy: 0.8743 - loss: 0.3555

 738/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 684us/step - accuracy: 0.8750 - loss: 0.3556

 811/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 685us/step - accuracy: 0.8757 - loss: 0.3542

 884/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 685us/step - accuracy: 0.8748 - loss: 0.3565

 960/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 684us/step - accuracy: 0.8756 - loss: 0.3536

1033/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 684us/step - accuracy: 0.8758 - loss: 0.3520

1108/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 683us/step - accuracy: 0.8764 - loss: 0.3513

1183/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 682us/step - accuracy: 0.8765 - loss: 0.3507

1259/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 681us/step - accuracy: 0.8760 - loss: 0.3519

1335/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 680us/step - accuracy: 0.8767 - loss: 0.3497

1411/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 679us/step - accuracy: 0.8772 - loss: 0.3487

1486/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 678us/step - accuracy: 0.8768 - loss: 0.3494

1559/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 679us/step - accuracy: 0.8766 - loss: 0.3491

1635/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 678us/step - accuracy: 0.8762 - loss: 0.3490

1710/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 678us/step - accuracy: 0.8758 - loss: 0.3491

1719/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 723us/step - accuracy: 0.8758 - loss: 0.3493 - val_accuracy: 0.8602 - val_loss: 0.3821

Epoch 9/30


   1/1719 ━━━━━━━━━━━━━━━━━━━━ 11s 7ms/step - accuracy: 0.8750 - loss: 0.3682

  72/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 710us/step - accuracy: 0.8819 - loss: 0.3228

 141/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 718us/step - accuracy: 0.8732 - loss: 0.3543

 210/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 723us/step - accuracy: 0.8757 - loss: 0.3480

 281/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 718us/step - accuracy: 0.8753 - loss: 0.3485

 354/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 713us/step - accuracy: 0.8769 - loss: 0.3498

 428/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 707us/step - accuracy: 0.8775 - loss: 0.3481

 503/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 702us/step - accuracy: 0.8764 - loss: 0.3474

 579/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 697us/step - accuracy: 0.8771 - loss: 0.3457

 656/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 692us/step - accuracy: 0.8779 - loss: 0.3441

 732/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 689us/step - accuracy: 0.8784 - loss: 0.3445

 807/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 688us/step - accuracy: 0.8789 - loss: 0.3437

 884/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 685us/step - accuracy: 0.8782 - loss: 0.3455

 960/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 683us/step - accuracy: 0.8791 - loss: 0.3426

1036/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 682us/step - accuracy: 0.8795 - loss: 0.3412

1111/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 681us/step - accuracy: 0.8798 - loss: 0.3408

1185/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 681us/step - accuracy: 0.8797 - loss: 0.3399

1260/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 680us/step - accuracy: 0.8794 - loss: 0.3411

1335/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 680us/step - accuracy: 0.8801 - loss: 0.3390

1412/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 678us/step - accuracy: 0.8804 - loss: 0.3380

1489/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 677us/step - accuracy: 0.8800 - loss: 0.3387

1565/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 676us/step - accuracy: 0.8798 - loss: 0.3388

1640/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 676us/step - accuracy: 0.8795 - loss: 0.3386

1715/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 676us/step - accuracy: 0.8792 - loss: 0.3390

1719/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 725us/step - accuracy: 0.8792 - loss: 0.3388 - val_accuracy: 0.8638 - val_loss: 0.3761

Epoch 10/30


   1/1719 ━━━━━━━━━━━━━━━━━━━━ 13s 8ms/step - accuracy: 0.8750 - loss: 0.3566

  74/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 694us/step - accuracy: 0.8847 - loss: 0.3174

 122/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 842us/step - accuracy: 0.8770 - loss: 0.3439

 178/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 860us/step - accuracy: 0.8773 - loss: 0.3388

 251/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 810us/step - accuracy: 0.8810 - loss: 0.3361

 323/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 784us/step - accuracy: 0.8799 - loss: 0.3370

 396/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 767us/step - accuracy: 0.8790 - loss: 0.3403

 471/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 751us/step - accuracy: 0.8792 - loss: 0.3383

 546/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 740us/step - accuracy: 0.8793 - loss: 0.3378

 621/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 732us/step - accuracy: 0.8804 - loss: 0.3350

 696/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 725us/step - accuracy: 0.8811 - loss: 0.3339

 766/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 724us/step - accuracy: 0.8810 - loss: 0.3341

 838/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 722us/step - accuracy: 0.8813 - loss: 0.3359

 911/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 719us/step - accuracy: 0.8815 - loss: 0.3338

 987/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 715us/step - accuracy: 0.8821 - loss: 0.3322

1060/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 713us/step - accuracy: 0.8823 - loss: 0.3310

1134/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 711us/step - accuracy: 0.8830 - loss: 0.3300

1209/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 708us/step - accuracy: 0.8827 - loss: 0.3310

1283/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 706us/step - accuracy: 0.8827 - loss: 0.3303

1355/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 706us/step - accuracy: 0.8832 - loss: 0.3294

1429/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 705us/step - accuracy: 0.8833 - loss: 0.3289

1502/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 704us/step - accuracy: 0.8831 - loss: 0.3290

1575/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 703us/step - accuracy: 0.8827 - loss: 0.3295

1650/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 702us/step - accuracy: 0.8823 - loss: 0.3296

1719/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 745us/step - accuracy: 0.8822 - loss: 0.3295 - val_accuracy: 0.8670 - val_loss: 0.3713

Epoch 11/30


   1/1719 ━━━━━━━━━━━━━━━━━━━━ 13s 8ms/step - accuracy: 0.9062 - loss: 0.3385

  73/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 697us/step - accuracy: 0.8887 - loss: 0.3055

 146/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 696us/step - accuracy: 0.8780 - loss: 0.3356

 207/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 735us/step - accuracy: 0.8806 - loss: 0.3299

 272/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 744us/step - accuracy: 0.8811 - loss: 0.3300

 320/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 791us/step - accuracy: 0.8816 - loss: 0.3289

 373/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 815us/step - accuracy: 0.8818 - loss: 0.3302

 417/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 853us/step - accuracy: 0.8816 - loss: 0.3306

 449/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 925us/step - accuracy: 0.8815 - loss: 0.3305

 460/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 1ms/step - accuracy: 0.8819 - loss: 0.3293  

 497/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 1ms/step - accuracy: 0.8812 - loss: 0.3290

 537/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 1ms/step - accuracy: 0.8818 - loss: 0.3283

 552/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 1ms/step - accuracy: 0.8817 - loss: 0.3281

 598/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 1ms/step - accuracy: 0.8824 - loss: 0.3258

 644/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 1ms/step - accuracy: 0.8833 - loss: 0.3246

 692/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 1ms/step - accuracy: 0.8835 - loss: 0.3250

 742/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 1ms/step - accuracy: 0.8834 - loss: 0.3255

 755/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 1ms/step - accuracy: 0.8836 - loss: 0.3253

 798/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 1ms/step - accuracy: 0.8847 - loss: 0.3245

 847/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 1ms/step - accuracy: 0.8837 - loss: 0.3273

 883/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - accuracy: 0.8839 - loss: 0.3264

 928/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - accuracy: 0.8846 - loss: 0.3241

 983/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - accuracy: 0.8849 - loss: 0.3236

1047/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - accuracy: 0.8848 - loss: 0.3226

1110/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - accuracy: 0.8852 - loss: 0.3222

1175/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - accuracy: 0.8852 - loss: 0.3218

1241/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - accuracy: 0.8849 - loss: 0.3225

1307/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - accuracy: 0.8855 - loss: 0.3212

1374/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - accuracy: 0.8858 - loss: 0.3203

1439/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - accuracy: 0.8859 - loss: 0.3204

1504/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - accuracy: 0.8857 - loss: 0.3205

1564/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - accuracy: 0.8854 - loss: 0.3207

1613/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - accuracy: 0.8853 - loss: 0.3206

1651/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - accuracy: 0.8848 - loss: 0.3212

1668/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - accuracy: 0.8847 - loss: 0.3216

1709/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - accuracy: 0.8848 - loss: 0.3209

1719/1719 ━━━━━━━━━━━━━━━━━━━━ 2s 1ms/step - accuracy: 0.8847 - loss: 0.3210 - val_accuracy: 0.8692 - val_loss: 0.3669

Epoch 12/30


   1/1719 ━━━━━━━━━━━━━━━━━━━━ 17s 10ms/step - accuracy: 0.9062 - loss: 0.3225

  17/1719 ━━━━━━━━━━━━━━━━━━━━ 5s 3ms/step - accuracy: 0.9007 - loss: 0.2774  

  48/1719 ━━━━━━━━━━━━━━━━━━━━ 3s 2ms/step - accuracy: 0.8926 - loss: 0.2927

  94/1719 ━━━━━━━━━━━━━━━━━━━━ 2s 2ms/step - accuracy: 0.8866 - loss: 0.3101

 142/1719 ━━━━━━━━━━━━━━━━━━━━ 2s 1ms/step - accuracy: 0.8814 - loss: 0.3280

 194/1719 ━━━━━━━━━━━━━━━━━━━━ 2s 1ms/step - accuracy: 0.8847 - loss: 0.3224

 256/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 1ms/step - accuracy: 0.8857 - loss: 0.3199

 324/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 1ms/step - accuracy: 0.8856 - loss: 0.3197

 380/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 1ms/step - accuracy: 0.8843 - loss: 0.3246

 437/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 1ms/step - accuracy: 0.8850 - loss: 0.3226

 496/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 1ms/step - accuracy: 0.8848 - loss: 0.3209

 558/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 997us/step - accuracy: 0.8853 - loss: 0.3197

 623/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 975us/step - accuracy: 0.8860 - loss: 0.3176

 693/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 948us/step - accuracy: 0.8864 - loss: 0.3173

 765/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 925us/step - accuracy: 0.8868 - loss: 0.3170

 835/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 908us/step - accuracy: 0.8869 - loss: 0.3188

 904/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 895us/step - accuracy: 0.8871 - loss: 0.3176

 974/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 882us/step - accuracy: 0.8879 - loss: 0.3155

1044/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 871us/step - accuracy: 0.8879 - loss: 0.3146

1116/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 859us/step - accuracy: 0.8882 - loss: 0.3143

1189/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 849us/step - accuracy: 0.8881 - loss: 0.3144

1264/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 838us/step - accuracy: 0.8882 - loss: 0.3146

1340/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 829us/step - accuracy: 0.8888 - loss: 0.3129

1414/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 821us/step - accuracy: 0.8891 - loss: 0.3119

1488/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 814us/step - accuracy: 0.8887 - loss: 0.3128

1562/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 808us/step - accuracy: 0.8886 - loss: 0.3129

1634/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 803us/step - accuracy: 0.8883 - loss: 0.3129

1705/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 799us/step - accuracy: 0.8880 - loss: 0.3132

1719/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 845us/step - accuracy: 0.8879 - loss: 0.3132 - val_accuracy: 0.8698 - val_loss: 0.3634

Epoch 13/30


   1/1719 ━━━━━━━━━━━━━━━━━━━━ 13s 8ms/step - accuracy: 0.9062 - loss: 0.3159

  73/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 701us/step - accuracy: 0.8951 - loss: 0.2897

 145/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 702us/step - accuracy: 0.8858 - loss: 0.3194

 218/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 698us/step - accuracy: 0.8883 - loss: 0.3135

 292/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 693us/step - accuracy: 0.8881 - loss: 0.3134

 362/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 697us/step - accuracy: 0.8892 - loss: 0.3142

 430/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 703us/step - accuracy: 0.8891 - loss: 0.3135

 505/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 699us/step - accuracy: 0.8882 - loss: 0.3133

 576/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 700us/step - accuracy: 0.8889 - loss: 0.3110

 652/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 696us/step - accuracy: 0.8896 - loss: 0.3091

 725/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 696us/step - accuracy: 0.8901 - loss: 0.3096

 798/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 695us/step - accuracy: 0.8909 - loss: 0.3087

 872/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 694us/step - accuracy: 0.8901 - loss: 0.3109

 944/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 694us/step - accuracy: 0.8905 - loss: 0.3085

1019/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 692us/step - accuracy: 0.8910 - loss: 0.3067

1096/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 689us/step - accuracy: 0.8913 - loss: 0.3060

1170/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 689us/step - accuracy: 0.8913 - loss: 0.3062

1243/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 689us/step - accuracy: 0.8908 - loss: 0.3074

1316/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 689us/step - accuracy: 0.8914 - loss: 0.3056

1389/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 689us/step - accuracy: 0.8915 - loss: 0.3049

1462/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 689us/step - accuracy: 0.8915 - loss: 0.3051

1536/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 688us/step - accuracy: 0.8913 - loss: 0.3051

1608/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 689us/step - accuracy: 0.8912 - loss: 0.3054

1681/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 689us/step - accuracy: 0.8907 - loss: 0.3060

1719/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 736us/step - accuracy: 0.8907 - loss: 0.3058 - val_accuracy: 0.8698 - val_loss: 0.3605

Epoch 14/30


   1/1719 ━━━━━━━━━━━━━━━━━━━━ 14s 8ms/step - accuracy: 0.9062 - loss: 0.2995

  72/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 713us/step - accuracy: 0.8984 - loss: 0.2815

 145/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 702us/step - accuracy: 0.8881 - loss: 0.3122

 218/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 697us/step - accuracy: 0.8903 - loss: 0.3064

 292/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 693us/step - accuracy: 0.8906 - loss: 0.3062

 366/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 690us/step - accuracy: 0.8912 - loss: 0.3068

 441/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 687us/step - accuracy: 0.8904 - loss: 0.3076

 516/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 685us/step - accuracy: 0.8901 - loss: 0.3059

 589/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 686us/step - accuracy: 0.8906 - loss: 0.3038

 663/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 685us/step - accuracy: 0.8913 - loss: 0.3022

 735/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 686us/step - accuracy: 0.8914 - loss: 0.3030

 810/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 685us/step - accuracy: 0.8922 - loss: 0.3018

 886/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 683us/step - accuracy: 0.8915 - loss: 0.3038

 959/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 684us/step - accuracy: 0.8920 - loss: 0.3013

1032/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 684us/step - accuracy: 0.8923 - loss: 0.2999

1105/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 685us/step - accuracy: 0.8926 - loss: 0.2996

1180/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 684us/step - accuracy: 0.8929 - loss: 0.2990

1255/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 683us/step - accuracy: 0.8925 - loss: 0.3001

1329/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 683us/step - accuracy: 0.8932 - loss: 0.2986

1404/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 682us/step - accuracy: 0.8934 - loss: 0.2979

1480/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 681us/step - accuracy: 0.8932 - loss: 0.2985

1553/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 681us/step - accuracy: 0.8932 - loss: 0.2982

1630/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 680us/step - accuracy: 0.8927 - loss: 0.2988

1707/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 679us/step - accuracy: 0.8924 - loss: 0.2989

1719/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 722us/step - accuracy: 0.8923 - loss: 0.2990 - val_accuracy: 0.8710 - val_loss: 0.3577

Epoch 15/30


   1/1719 ━━━━━━━━━━━━━━━━━━━━ 12s 7ms/step - accuracy: 0.9062 - loss: 0.2889

  74/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 695us/step - accuracy: 0.9024 - loss: 0.2782

 149/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 683us/step - accuracy: 0.8905 - loss: 0.3058

 225/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 675us/step - accuracy: 0.8928 - loss: 0.3001

 300/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 675us/step - accuracy: 0.8921 - loss: 0.2985

 378/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 669us/step - accuracy: 0.8912 - loss: 0.3026

 455/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 666us/step - accuracy: 0.8923 - loss: 0.2996

 532/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 664us/step - accuracy: 0.8920 - loss: 0.2987

 609/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 662us/step - accuracy: 0.8929 - loss: 0.2958

 687/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 661us/step - accuracy: 0.8931 - loss: 0.2957

 757/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 666us/step - accuracy: 0.8935 - loss: 0.2959

 826/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 671us/step - accuracy: 0.8935 - loss: 0.2964

 890/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 680us/step - accuracy: 0.8932 - loss: 0.2971

 959/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 684us/step - accuracy: 0.8937 - loss: 0.2947

1030/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 686us/step - accuracy: 0.8941 - loss: 0.2933

1101/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 687us/step - accuracy: 0.8946 - loss: 0.2925

1171/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 689us/step - accuracy: 0.8946 - loss: 0.2926

1244/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 689us/step - accuracy: 0.8939 - loss: 0.2938

1317/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 689us/step - accuracy: 0.8949 - loss: 0.2922

1390/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 689us/step - accuracy: 0.8951 - loss: 0.2915

1463/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 689us/step - accuracy: 0.8950 - loss: 0.2918

1536/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 689us/step - accuracy: 0.8948 - loss: 0.2918

1607/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 690us/step - accuracy: 0.8946 - loss: 0.2922

1681/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 689us/step - accuracy: 0.8941 - loss: 0.2929

1719/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 736us/step - accuracy: 0.8941 - loss: 0.2926 - val_accuracy: 0.8714 - val_loss: 0.3551

Epoch 16/30


   1/1719 ━━━━━━━━━━━━━━━━━━━━ 13s 8ms/step - accuracy: 0.9062 - loss: 0.2809

  72/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 704us/step - accuracy: 0.9028 - loss: 0.2685

 145/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 695us/step - accuracy: 0.8922 - loss: 0.2984

 218/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 694us/step - accuracy: 0.8936 - loss: 0.2932

 293/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 689us/step - accuracy: 0.8937 - loss: 0.2930

 367/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 687us/step - accuracy: 0.8944 - loss: 0.2944

 440/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 688us/step - accuracy: 0.8942 - loss: 0.2944

 512/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 689us/step - accuracy: 0.8941 - loss: 0.2925

 582/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 692us/step - accuracy: 0.8947 - loss: 0.2908

 654/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 693us/step - accuracy: 0.8955 - loss: 0.2889

 727/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 692us/step - accuracy: 0.8959 - loss: 0.2894

 800/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 692us/step - accuracy: 0.8964 - loss: 0.2887

 874/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 691us/step - accuracy: 0.8956 - loss: 0.2908

 947/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 691us/step - accuracy: 0.8960 - loss: 0.2883

1022/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 690us/step - accuracy: 0.8963 - loss: 0.2871

1094/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 690us/step - accuracy: 0.8970 - loss: 0.2859

1168/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 689us/step - accuracy: 0.8968 - loss: 0.2866

1240/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 690us/step - accuracy: 0.8966 - loss: 0.2870

1314/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 690us/step - accuracy: 0.8973 - loss: 0.2858

1388/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 689us/step - accuracy: 0.8973 - loss: 0.2853

1463/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 688us/step - accuracy: 0.8973 - loss: 0.2856

1533/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 689us/step - accuracy: 0.8969 - loss: 0.2857

1606/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 689us/step - accuracy: 0.8969 - loss: 0.2860

1680/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 689us/step - accuracy: 0.8964 - loss: 0.2867

1719/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 734us/step - accuracy: 0.8964 - loss: 0.2864 - val_accuracy: 0.8730 - val_loss: 0.3530

Epoch 17/30


   1/1719 ━━━━━━━━━━━━━━━━━━━━ 12s 7ms/step - accuracy: 0.9375 - loss: 0.2681

  72/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 712us/step - accuracy: 0.9067 - loss: 0.2624

 146/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 698us/step - accuracy: 0.8943 - loss: 0.2926

 221/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 688us/step - accuracy: 0.8952 - loss: 0.2885

 297/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 682us/step - accuracy: 0.8956 - loss: 0.2862

 373/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 678us/step - accuracy: 0.8959 - loss: 0.2880

 449/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 675us/step - accuracy: 0.8960 - loss: 0.2881

 523/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 675us/step - accuracy: 0.8961 - loss: 0.2869

 599/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 674us/step - accuracy: 0.8973 - loss: 0.2834

 673/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 675us/step - accuracy: 0.8979 - loss: 0.2832

 746/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 677us/step - accuracy: 0.8978 - loss: 0.2835

 818/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 679us/step - accuracy: 0.8980 - loss: 0.2833

 886/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 683us/step - accuracy: 0.8978 - loss: 0.2846

 958/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 685us/step - accuracy: 0.8985 - loss: 0.2824

1029/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 686us/step - accuracy: 0.8986 - loss: 0.2810

1101/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 687us/step - accuracy: 0.8992 - loss: 0.2801

1173/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 688us/step - accuracy: 0.8991 - loss: 0.2805

1245/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 689us/step - accuracy: 0.8986 - loss: 0.2814

1315/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 690us/step - accuracy: 0.8995 - loss: 0.2798

1385/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 692us/step - accuracy: 0.8995 - loss: 0.2796

1456/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 693us/step - accuracy: 0.8996 - loss: 0.2796

1528/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 693us/step - accuracy: 0.8992 - loss: 0.2799

1598/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 694us/step - accuracy: 0.8991 - loss: 0.2802

1671/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 694us/step - accuracy: 0.8986 - loss: 0.2811

1719/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 742us/step - accuracy: 0.8987 - loss: 0.2806 - val_accuracy: 0.8740 - val_loss: 0.3507

Epoch 18/30


   1/1719 ━━━━━━━━━━━━━━━━━━━━ 12s 7ms/step - accuracy: 0.9062 - loss: 0.2561

  32/1719 ━━━━━━━━━━━━━━━━━━━━ 2s 2ms/step - accuracy: 0.9072 - loss: 0.2552 

  97/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 1ms/step - accuracy: 0.9008 - loss: 0.2727

 168/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 908us/step - accuracy: 0.8960 - loss: 0.2842

 240/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 845us/step - accuracy: 0.9001 - loss: 0.2792

 309/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 820us/step - accuracy: 0.8994 - loss: 0.2782

 378/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 803us/step - accuracy: 0.8984 - loss: 0.2839

 448/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 789us/step - accuracy: 0.8985 - loss: 0.2822

 520/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 777us/step - accuracy: 0.8992 - loss: 0.2807

 593/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 766us/step - accuracy: 0.8998 - loss: 0.2785

 662/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 762us/step - accuracy: 0.9006 - loss: 0.2772

 730/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 760us/step - accuracy: 0.9009 - loss: 0.2772

 797/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 760us/step - accuracy: 0.9012 - loss: 0.2770

 868/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 756us/step - accuracy: 0.9004 - loss: 0.2794

 942/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 750us/step - accuracy: 0.9011 - loss: 0.2765

1011/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 749us/step - accuracy: 0.9013 - loss: 0.2754

1082/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 746us/step - accuracy: 0.9018 - loss: 0.2742

1151/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 745us/step - accuracy: 0.9015 - loss: 0.2748

1221/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 744us/step - accuracy: 0.9015 - loss: 0.2753

1292/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 741us/step - accuracy: 0.9019 - loss: 0.2743

1364/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 739us/step - accuracy: 0.9022 - loss: 0.2737

1437/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 737us/step - accuracy: 0.9022 - loss: 0.2740

1510/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 734us/step - accuracy: 0.9019 - loss: 0.2741

1581/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 733us/step - accuracy: 0.9016 - loss: 0.2748

1654/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 731us/step - accuracy: 0.9010 - loss: 0.2754

1719/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 778us/step - accuracy: 0.9011 - loss: 0.2751 - val_accuracy: 0.8752 - val_loss: 0.3486

Epoch 19/30


   1/1719 ━━━━━━━━━━━━━━━━━━━━ 15s 9ms/step - accuracy: 0.9062 - loss: 0.2479

  70/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 728us/step - accuracy: 0.9089 - loss: 0.2505

 140/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 722us/step - accuracy: 0.8996 - loss: 0.2802

 211/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 720us/step - accuracy: 0.8993 - loss: 0.2767

 281/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 720us/step - accuracy: 0.8994 - loss: 0.2751

 351/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 720us/step - accuracy: 0.9003 - loss: 0.2777

 421/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 720us/step - accuracy: 0.9012 - loss: 0.2771

 492/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 719us/step - accuracy: 0.9012 - loss: 0.2750

 564/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 717us/step - accuracy: 0.9015 - loss: 0.2729

 635/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 716us/step - accuracy: 0.9022 - loss: 0.2720

 703/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 718us/step - accuracy: 0.9027 - loss: 0.2714

 774/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 717us/step - accuracy: 0.9027 - loss: 0.2727

 844/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 717us/step - accuracy: 0.9021 - loss: 0.2742

 915/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 716us/step - accuracy: 0.9030 - loss: 0.2717

 989/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 713us/step - accuracy: 0.9035 - loss: 0.2705

1062/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 711us/step - accuracy: 0.9036 - loss: 0.2691

1139/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 707us/step - accuracy: 0.9037 - loss: 0.2691

1214/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 705us/step - accuracy: 0.9033 - loss: 0.2699

1290/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 702us/step - accuracy: 0.9038 - loss: 0.2690

1366/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 700us/step - accuracy: 0.9041 - loss: 0.2682

1441/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 698us/step - accuracy: 0.9041 - loss: 0.2687

1515/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 697us/step - accuracy: 0.9037 - loss: 0.2689

1591/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 696us/step - accuracy: 0.9036 - loss: 0.2693

1663/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 696us/step - accuracy: 0.9030 - loss: 0.2703

1719/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 742us/step - accuracy: 0.9031 - loss: 0.2698 - val_accuracy: 0.8764 - val_loss: 0.3466

Epoch 20/30


   1/1719 ━━━━━━━━━━━━━━━━━━━━ 12s 8ms/step - accuracy: 0.9375 - loss: 0.2366

  73/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 701us/step - accuracy: 0.9101 - loss: 0.2470

 148/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 685us/step - accuracy: 0.8995 - loss: 0.2749

 224/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 678us/step - accuracy: 0.9012 - loss: 0.2711

 300/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 674us/step - accuracy: 0.9010 - loss: 0.2684

 377/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 670us/step - accuracy: 0.9018 - loss: 0.2723

 452/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 670us/step - accuracy: 0.9027 - loss: 0.2709

 528/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 669us/step - accuracy: 0.9026 - loss: 0.2703

 605/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 667us/step - accuracy: 0.9037 - loss: 0.2669

 679/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 668us/step - accuracy: 0.9047 - loss: 0.2667

 752/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 670us/step - accuracy: 0.9045 - loss: 0.2670

 828/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 669us/step - accuracy: 0.9046 - loss: 0.2675

 901/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 671us/step - accuracy: 0.9046 - loss: 0.2676

 975/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 671us/step - accuracy: 0.9053 - loss: 0.2655

1050/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 671us/step - accuracy: 0.9051 - loss: 0.2646

1125/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 671us/step - accuracy: 0.9054 - loss: 0.2639

1199/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 672us/step - accuracy: 0.9049 - loss: 0.2650

1273/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 672us/step - accuracy: 0.9055 - loss: 0.2641

1347/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 673us/step - accuracy: 0.9059 - loss: 0.2635

1423/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 672us/step - accuracy: 0.9062 - loss: 0.2628

1495/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 674us/step - accuracy: 0.9056 - loss: 0.2640

1568/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 674us/step - accuracy: 0.9052 - loss: 0.2643

1639/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 676us/step - accuracy: 0.9050 - loss: 0.2644

1713/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 676us/step - accuracy: 0.9049 - loss: 0.2647

1719/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 722us/step - accuracy: 0.9050 - loss: 0.2647 - val_accuracy: 0.8766 - val_loss: 0.3473

Epoch 21/30


   1/1719 ━━━━━━━━━━━━━━━━━━━━ 12s 7ms/step - accuracy: 0.9375 - loss: 0.2353

  75/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 684us/step - accuracy: 0.9121 - loss: 0.2454

 149/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 682us/step - accuracy: 0.9018 - loss: 0.2702

 224/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 677us/step - accuracy: 0.9030 - loss: 0.2663

 297/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 680us/step - accuracy: 0.9036 - loss: 0.2643

 373/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 676us/step - accuracy: 0.9042 - loss: 0.2664

 446/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 678us/step - accuracy: 0.9049 - loss: 0.2660

 516/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 683us/step - accuracy: 0.9053 - loss: 0.2654

 590/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 683us/step - accuracy: 0.9059 - loss: 0.2630

 665/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 682us/step - accuracy: 0.9071 - loss: 0.2618

 738/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 682us/step - accuracy: 0.9065 - loss: 0.2625

 813/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 681us/step - accuracy: 0.9072 - loss: 0.2612

 887/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 681us/step - accuracy: 0.9065 - loss: 0.2634

 963/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 680us/step - accuracy: 0.9072 - loss: 0.2609

1035/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 681us/step - accuracy: 0.9071 - loss: 0.2598

1109/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 681us/step - accuracy: 0.9075 - loss: 0.2594

1183/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 681us/step - accuracy: 0.9074 - loss: 0.2590

1259/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 680us/step - accuracy: 0.9071 - loss: 0.2599

1335/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 678us/step - accuracy: 0.9079 - loss: 0.2584

1406/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 680us/step - accuracy: 0.9080 - loss: 0.2583

1478/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 681us/step - accuracy: 0.9074 - loss: 0.2591

1554/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 680us/step - accuracy: 0.9073 - loss: 0.2590

1630/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 680us/step - accuracy: 0.9069 - loss: 0.2596

1701/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 681us/step - accuracy: 0.9069 - loss: 0.2599

1719/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 724us/step - accuracy: 0.9068 - loss: 0.2598 - val_accuracy: 0.8780 - val_loss: 0.3465

Epoch 22/30


   1/1719 ━━━━━━━━━━━━━━━━━━━━ 12s 7ms/step - accuracy: 0.9375 - loss: 0.2248

  77/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 662us/step - accuracy: 0.9123 - loss: 0.2391

 152/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 668us/step - accuracy: 0.9028 - loss: 0.2652

 227/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 668us/step - accuracy: 0.9051 - loss: 0.2609

 304/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 666us/step - accuracy: 0.9051 - loss: 0.2587

 380/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 665us/step - accuracy: 0.9053 - loss: 0.2635

 454/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 666us/step - accuracy: 0.9067 - loss: 0.2605

 528/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 668us/step - accuracy: 0.9070 - loss: 0.2604

 604/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 667us/step - accuracy: 0.9080 - loss: 0.2573

 678/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 669us/step - accuracy: 0.9092 - loss: 0.2570

 752/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 670us/step - accuracy: 0.9088 - loss: 0.2572

 826/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 671us/step - accuracy: 0.9087 - loss: 0.2579

 900/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 671us/step - accuracy: 0.9087 - loss: 0.2579

 977/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 670us/step - accuracy: 0.9094 - loss: 0.2557

1052/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 670us/step - accuracy: 0.9093 - loss: 0.2547

1129/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 669us/step - accuracy: 0.9095 - loss: 0.2540

1205/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 669us/step - accuracy: 0.9089 - loss: 0.2550

1281/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 668us/step - accuracy: 0.9092 - loss: 0.2545

1357/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 668us/step - accuracy: 0.9098 - loss: 0.2537

1434/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 667us/step - accuracy: 0.9097 - loss: 0.2538

1510/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 667us/step - accuracy: 0.9093 - loss: 0.2541

1587/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 667us/step - accuracy: 0.9090 - loss: 0.2545

1662/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 667us/step - accuracy: 0.9084 - loss: 0.2557

1719/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 711us/step - accuracy: 0.9086 - loss: 0.2550 - val_accuracy: 0.8782 - val_loss: 0.3464

Epoch 23/30


   1/1719 ━━━━━━━━━━━━━━━━━━━━ 12s 7ms/step - accuracy: 0.9375 - loss: 0.2275

  75/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 680us/step - accuracy: 0.9146 - loss: 0.2359

 146/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 693us/step - accuracy: 0.9050 - loss: 0.2613

 220/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 688us/step - accuracy: 0.9057 - loss: 0.2583

 294/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 686us/step - accuracy: 0.9066 - loss: 0.2547

 369/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 682us/step - accuracy: 0.9073 - loss: 0.2569

 444/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 681us/step - accuracy: 0.9076 - loss: 0.2572

 518/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 681us/step - accuracy: 0.9087 - loss: 0.2558

 590/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 683us/step - accuracy: 0.9096 - loss: 0.2535

 664/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 683us/step - accuracy: 0.9106 - loss: 0.2525

 739/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 682us/step - accuracy: 0.9101 - loss: 0.2531

 813/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 682us/step - accuracy: 0.9107 - loss: 0.2520

 889/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 680us/step - accuracy: 0.9101 - loss: 0.2540

 965/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 679us/step - accuracy: 0.9109 - loss: 0.2513

1042/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 677us/step - accuracy: 0.9107 - loss: 0.2504

1118/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 676us/step - accuracy: 0.9110 - loss: 0.2499

1194/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 675us/step - accuracy: 0.9104 - loss: 0.2507

1268/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 676us/step - accuracy: 0.9107 - loss: 0.2501

1343/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 675us/step - accuracy: 0.9112 - loss: 0.2492

1417/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 676us/step - accuracy: 0.9115 - loss: 0.2485

1492/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 675us/step - accuracy: 0.9108 - loss: 0.2497

1568/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 675us/step - accuracy: 0.9105 - loss: 0.2501

1638/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 676us/step - accuracy: 0.9102 - loss: 0.2503

1713/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 676us/step - accuracy: 0.9101 - loss: 0.2506

1719/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 720us/step - accuracy: 0.9102 - loss: 0.2505 - val_accuracy: 0.8770 - val_loss: 0.3461

Epoch 24/30


   1/1719 ━━━━━━━━━━━━━━━━━━━━ 12s 7ms/step - accuracy: 0.9375 - loss: 0.2132

  72/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 709us/step - accuracy: 0.9167 - loss: 0.2284

 146/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 693us/step - accuracy: 0.9062 - loss: 0.2566

 222/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 684us/step - accuracy: 0.9084 - loss: 0.2528

 299/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 677us/step - accuracy: 0.9087 - loss: 0.2497

 375/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 674us/step - accuracy: 0.9091 - loss: 0.2532

 450/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 674us/step - accuracy: 0.9097 - loss: 0.2524

 527/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 670us/step - accuracy: 0.9108 - loss: 0.2514

 603/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 669us/step - accuracy: 0.9117 - loss: 0.2485

 680/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 668us/step - accuracy: 0.9126 - loss: 0.2479

 757/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 666us/step - accuracy: 0.9122 - loss: 0.2486

 834/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 665us/step - accuracy: 0.9117 - loss: 0.2496

 909/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 666us/step - accuracy: 0.9122 - loss: 0.2484

 984/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 666us/step - accuracy: 0.9126 - loss: 0.2467

1054/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 670us/step - accuracy: 0.9126 - loss: 0.2455

1124/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 673us/step - accuracy: 0.9128 - loss: 0.2452

1194/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 676us/step - accuracy: 0.9121 - loss: 0.2462

1265/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 677us/step - accuracy: 0.9124 - loss: 0.2457

1339/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 677us/step - accuracy: 0.9130 - loss: 0.2447

1415/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 676us/step - accuracy: 0.9132 - loss: 0.2440

1488/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 677us/step - accuracy: 0.9125 - loss: 0.2452

1561/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 678us/step - accuracy: 0.9123 - loss: 0.2454

1635/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 678us/step - accuracy: 0.9120 - loss: 0.2457

1707/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 679us/step - accuracy: 0.9119 - loss: 0.2461

1719/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 725us/step - accuracy: 0.9119 - loss: 0.2460 - val_accuracy: 0.8760 - val_loss: 0.3444

Epoch 25/30


   1/1719 ━━━━━━━━━━━━━━━━━━━━ 13s 8ms/step - accuracy: 0.9375 - loss: 0.2070

  74/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 694us/step - accuracy: 0.9198 - loss: 0.2264

 150/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 677us/step - accuracy: 0.9100 - loss: 0.2510

 223/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 682us/step - accuracy: 0.9113 - loss: 0.2477

 297/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 682us/step - accuracy: 0.9110 - loss: 0.2457

 371/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 681us/step - accuracy: 0.9116 - loss: 0.2478

 443/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 684us/step - accuracy: 0.9116 - loss: 0.2487

 515/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 686us/step - accuracy: 0.9124 - loss: 0.2476

 588/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 687us/step - accuracy: 0.9133 - loss: 0.2451

 662/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 686us/step - accuracy: 0.9142 - loss: 0.2436

 735/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 686us/step - accuracy: 0.9135 - loss: 0.2448

 809/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 685us/step - accuracy: 0.9139 - loss: 0.2440

 881/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 686us/step - accuracy: 0.9136 - loss: 0.2454

 954/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 686us/step - accuracy: 0.9142 - loss: 0.2429

1027/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 686us/step - accuracy: 0.9140 - loss: 0.2420

1099/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 687us/step - accuracy: 0.9148 - loss: 0.2407

1172/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 687us/step - accuracy: 0.9142 - loss: 0.2411

1244/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 688us/step - accuracy: 0.9139 - loss: 0.2420

1318/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 687us/step - accuracy: 0.9147 - loss: 0.2407

1393/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 686us/step - accuracy: 0.9148 - loss: 0.2402

1465/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 687us/step - accuracy: 0.9145 - loss: 0.2409

1539/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 687us/step - accuracy: 0.9144 - loss: 0.2409

1614/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 686us/step - accuracy: 0.9140 - loss: 0.2413

1690/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 685us/step - accuracy: 0.9137 - loss: 0.2420

1719/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 728us/step - accuracy: 0.9137 - loss: 0.2418 - val_accuracy: 0.8758 - val_loss: 0.3456

Epoch 26/30


   1/1719 ━━━━━━━━━━━━━━━━━━━━ 12s 7ms/step - accuracy: 0.9375 - loss: 0.2077

  76/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 674us/step - accuracy: 0.9227 - loss: 0.2217

 151/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 674us/step - accuracy: 0.9116 - loss: 0.2473

 227/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 669us/step - accuracy: 0.9142 - loss: 0.2435

 301/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 672us/step - accuracy: 0.9133 - loss: 0.2405

 375/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 674us/step - accuracy: 0.9129 - loss: 0.2446

 447/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 677us/step - accuracy: 0.9133 - loss: 0.2438

 518/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 681us/step - accuracy: 0.9139 - loss: 0.2431

 593/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 680us/step - accuracy: 0.9147 - loss: 0.2404

 664/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 684us/step - accuracy: 0.9153 - loss: 0.2396

 731/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 689us/step - accuracy: 0.9151 - loss: 0.2400

 803/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 690us/step - accuracy: 0.9148 - loss: 0.2400

 874/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 691us/step - accuracy: 0.9147 - loss: 0.2412

 947/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 691us/step - accuracy: 0.9154 - loss: 0.2386

1020/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 691us/step - accuracy: 0.9155 - loss: 0.2370

1094/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 690us/step - accuracy: 0.9160 - loss: 0.2362

1169/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 689us/step - accuracy: 0.9154 - loss: 0.2368

1246/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 687us/step - accuracy: 0.9153 - loss: 0.2374

1323/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 685us/step - accuracy: 0.9160 - loss: 0.2364

1398/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 684us/step - accuracy: 0.9160 - loss: 0.2360

1472/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 684us/step - accuracy: 0.9156 - loss: 0.2366

1546/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 684us/step - accuracy: 0.9155 - loss: 0.2368

1621/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 683us/step - accuracy: 0.9152 - loss: 0.2371

1697/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 682us/step - accuracy: 0.9149 - loss: 0.2379

1719/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 726us/step - accuracy: 0.9150 - loss: 0.2376 - val_accuracy: 0.8748 - val_loss: 0.3454

Epoch 27/30


   1/1719 ━━━━━━━━━━━━━━━━━━━━ 12s 7ms/step - accuracy: 0.9375 - loss: 0.1976

  77/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 665us/step - accuracy: 0.9253 - loss: 0.2172

 150/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 677us/step - accuracy: 0.9142 - loss: 0.2423

 226/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 673us/step - accuracy: 0.9159 - loss: 0.2398

 302/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 669us/step - accuracy: 0.9156 - loss: 0.2357

 373/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 677us/step - accuracy: 0.9152 - loss: 0.2397

 443/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 683us/step - accuracy: 0.9153 - loss: 0.2402

 516/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 684us/step - accuracy: 0.9162 - loss: 0.2391

 590/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 683us/step - accuracy: 0.9171 - loss: 0.2364

 662/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 685us/step - accuracy: 0.9178 - loss: 0.2351

 733/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 687us/step - accuracy: 0.9170 - loss: 0.2364

 802/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 691us/step - accuracy: 0.9170 - loss: 0.2358

 868/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 696us/step - accuracy: 0.9167 - loss: 0.2374

 921/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 713us/step - accuracy: 0.9175 - loss: 0.2349

 951/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 744us/step - accuracy: 0.9175 - loss: 0.2345

1016/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 746us/step - accuracy: 0.9175 - loss: 0.2331

1079/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 751us/step - accuracy: 0.9179 - loss: 0.2324

1104/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 780us/step - accuracy: 0.9179 - loss: 0.2328

1167/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 781us/step - accuracy: 0.9174 - loss: 0.2328

1232/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 780us/step - accuracy: 0.9173 - loss: 0.2330

1299/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 779us/step - accuracy: 0.9178 - loss: 0.2324

1369/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 776us/step - accuracy: 0.9181 - loss: 0.2315

1441/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 772us/step - accuracy: 0.9179 - loss: 0.2322

1515/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 767us/step - accuracy: 0.9174 - loss: 0.2326

1586/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 765us/step - accuracy: 0.9173 - loss: 0.2330

1656/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 763us/step - accuracy: 0.9168 - loss: 0.2340

1719/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 806us/step - accuracy: 0.9169 - loss: 0.2334 - val_accuracy: 0.8742 - val_loss: 0.3470

Epoch 28/30


   1/1719 ━━━━━━━━━━━━━━━━━━━━ 12s 7ms/step - accuracy: 0.9375 - loss: 0.1997

  75/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 678us/step - accuracy: 0.9287 - loss: 0.2141

 150/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 673us/step - accuracy: 0.9165 - loss: 0.2381

 215/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 703us/step - accuracy: 0.9176 - loss: 0.2354

 277/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 727us/step - accuracy: 0.9178 - loss: 0.2342

 329/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 766us/step - accuracy: 0.9185 - loss: 0.2324

 360/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 841us/step - accuracy: 0.9181 - loss: 0.2357

 424/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 834us/step - accuracy: 0.9189 - loss: 0.2355

 488/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 828us/step - accuracy: 0.9191 - loss: 0.2347

 523/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 872us/step - accuracy: 0.9188 - loss: 0.2350

 576/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 879us/step - accuracy: 0.9194 - loss: 0.2325

 648/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 859us/step - accuracy: 0.9203 - loss: 0.2306

 721/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 842us/step - accuracy: 0.9199 - loss: 0.2309

 796/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 826us/step - accuracy: 0.9193 - loss: 0.2317

 870/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 814us/step - accuracy: 0.9191 - loss: 0.2330

 943/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 804us/step - accuracy: 0.9196 - loss: 0.2306

1016/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 796us/step - accuracy: 0.9197 - loss: 0.2290

1091/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 787us/step - accuracy: 0.9202 - loss: 0.2279

1161/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 784us/step - accuracy: 0.9194 - loss: 0.2285

1222/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 786us/step - accuracy: 0.9192 - loss: 0.2289

1285/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 786us/step - accuracy: 0.9197 - loss: 0.2283

1346/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 788us/step - accuracy: 0.9199 - loss: 0.2281

1409/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 789us/step - accuracy: 0.9199 - loss: 0.2275

1472/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 789us/step - accuracy: 0.9194 - loss: 0.2284

1498/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 809us/step - accuracy: 0.9194 - loss: 0.2284

1555/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 812us/step - accuracy: 0.9193 - loss: 0.2286

1624/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 809us/step - accuracy: 0.9190 - loss: 0.2290

1696/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 804us/step - accuracy: 0.9187 - loss: 0.2296

1719/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 850us/step - accuracy: 0.9188 - loss: 0.2293 - val_accuracy: 0.8730 - val_loss: 0.3447

Epoch 29/30


   1/1719 ━━━━━━━━━━━━━━━━━━━━ 13s 8ms/step - accuracy: 0.9375 - loss: 0.1830

  71/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 721us/step - accuracy: 0.9287 - loss: 0.2075

 145/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 701us/step - accuracy: 0.9181 - loss: 0.2348

 217/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 701us/step - accuracy: 0.9188 - loss: 0.2316

 292/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 693us/step - accuracy: 0.9190 - loss: 0.2290

 367/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 688us/step - accuracy: 0.9186 - loss: 0.2318

 437/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 693us/step - accuracy: 0.9191 - loss: 0.2322

 501/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 706us/step - accuracy: 0.9188 - loss: 0.2315

 563/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 718us/step - accuracy: 0.9201 - loss: 0.2284

 626/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 726us/step - accuracy: 0.9205 - loss: 0.2276

 653/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 775us/step - accuracy: 0.9206 - loss: 0.2270

 710/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 783us/step - accuracy: 0.9205 - loss: 0.2276

 774/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 784us/step - accuracy: 0.9200 - loss: 0.2284

 838/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 784us/step - accuracy: 0.9199 - loss: 0.2294

 907/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 779us/step - accuracy: 0.9204 - loss: 0.2283

 977/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 775us/step - accuracy: 0.9208 - loss: 0.2263

1048/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 770us/step - accuracy: 0.9208 - loss: 0.2252

1121/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 765us/step - accuracy: 0.9208 - loss: 0.2250

1194/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 760us/step - accuracy: 0.9202 - loss: 0.2256

1269/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 755us/step - accuracy: 0.9207 - loss: 0.2249

1343/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 750us/step - accuracy: 0.9211 - loss: 0.2243

1420/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 745us/step - accuracy: 0.9212 - loss: 0.2237

1492/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 743us/step - accuracy: 0.9206 - loss: 0.2248

1565/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 741us/step - accuracy: 0.9203 - loss: 0.2253

1641/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 737us/step - accuracy: 0.9201 - loss: 0.2254

1716/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 734us/step - accuracy: 0.9199 - loss: 0.2256

1719/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 781us/step - accuracy: 0.9199 - loss: 0.2256 - val_accuracy: 0.8740 - val_loss: 0.3464

Epoch 30/30


   1/1719 ━━━━━━━━━━━━━━━━━━━━ 12s 8ms/step - accuracy: 0.9375 - loss: 0.1792

  74/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 692us/step - accuracy: 0.9303 - loss: 0.2062

 148/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 688us/step - accuracy: 0.9196 - loss: 0.2301

 219/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 695us/step - accuracy: 0.9192 - loss: 0.2275

 292/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 693us/step - accuracy: 0.9197 - loss: 0.2249

 366/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 690us/step - accuracy: 0.9198 - loss: 0.2270

 441/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 688us/step - accuracy: 0.9199 - loss: 0.2281

 518/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 682us/step - accuracy: 0.9199 - loss: 0.2271

 595/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 679us/step - accuracy: 0.9209 - loss: 0.2241

 672/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 676us/step - accuracy: 0.9214 - loss: 0.2236

 749/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 673us/step - accuracy: 0.9209 - loss: 0.2240

 826/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 671us/step - accuracy: 0.9211 - loss: 0.2247

 903/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 670us/step - accuracy: 0.9211 - loss: 0.2247

 979/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 669us/step - accuracy: 0.9218 - loss: 0.2224

1056/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 668us/step - accuracy: 0.9217 - loss: 0.2214

1129/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 669us/step - accuracy: 0.9219 - loss: 0.2207

1205/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 669us/step - accuracy: 0.9213 - loss: 0.2214

1275/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 672us/step - accuracy: 0.9219 - loss: 0.2208

1348/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 673us/step - accuracy: 0.9222 - loss: 0.2204

1420/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 674us/step - accuracy: 0.9223 - loss: 0.2198

1491/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 676us/step - accuracy: 0.9218 - loss: 0.2208

1565/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 676us/step - accuracy: 0.9214 - loss: 0.2214

1641/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 675us/step - accuracy: 0.9212 - loss: 0.2215

1712/1719 ━━━━━━━━━━━━━━━━━━━━ 0s 676us/step - accuracy: 0.9211 - loss: 0.2217

1719/1719 ━━━━━━━━━━━━━━━━━━━━ 1s 722us/step - accuracy: 0.9211 - loss: 0.2217 - val_accuracy: 0.8744 - val_loss: 0.3471

The model is provided with both a training set and a validation set. At each step, the model will report its performance on both sets. This will also allow to visualize the accuracy and loss curves on both sets (more later).

When calling the fit method in Keras (or similar frameworks), each step corresponds to the evaluation of a mini-batch. A mini-batch is a subset of the training data, and during each step, the model updates its weights based on the error calculated from this mini-batch.

An epoch is defined as one complete pass through the entire training dataset. During an epoch, the model processes multiple mini-batches until it has seen all the training data once. This process is repeated for a specified number of epochs to optimize the model’s performance.

Visualization

import pandas as pd 

pd.DataFrame(history.history).plot(
    figsize=(8, 5), xlim=[0, 29], ylim=[0, 1], grid=True, xlabel="Epoch",
    style=["r--", "r--.", "b-", "b-*"])
plt.legend(loc="lower left")  # extra code
plt.show()

Visualization

Evaluating the model on our test

model.evaluate(X_test, y_test)

Making predictions

X_new = X_test[:3]
y_proba = model.predict(X_new)
y_proba.round(2)

1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 18ms/step

1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 23ms/step

array([[0., 0., 0., 0., 0., 0., 0., 0., 0., 1.],
       [0., 0., 1., 0., 0., 0., 0., 0., 0., 0.],
       [0., 1., 0., 0., 0., 0., 0., 0., 0., 0.]], dtype=float32)

y_pred = y_proba.argmax(axis=-1).astype(int)
y_pred

y_new = y_test[:3]
y_new

As can be seen, the predictions are unambiguous, with only one class per prediction exhibiting a high value.

Predicted vs Observed

plt.figure(figsize=(7.2, 2.4))
for index, image in enumerate(X_new):
    plt.subplot(1, 3, index + 1)
    plt.imshow(image, cmap="binary", interpolation="nearest")
    plt.axis('off')
    plt.title(class_names[y_test[index]])
plt.subplots_adjust(wspace=0.2, hspace=0.5)
plt.show()

Test Set Performance

from sklearn.metrics import classification_report

y_proba = model.predict(X_test)
y_pred = y_proba.argmax(axis=-1).astype(int)

Test Set Performance

print(classification_report(y_test, y_pred))

              precision    recall  f1-score   support

           0       0.83      0.85      0.84      1000
           1       0.95      0.98      0.96      1000
           2       0.81      0.73      0.77      1000
           3       0.91      0.83      0.87      1000
           4       0.65      0.92      0.76      1000
           5       0.93      0.97      0.95      1000
           6       0.81      0.53      0.64      1000
           7       0.97      0.84      0.90      1000
           8       0.93      0.98      0.95      1000
           9       0.90      0.98      0.94      1000

    accuracy                           0.86     10000
   macro avg       0.87      0.86      0.86     10000
weighted avg       0.87      0.86      0.86     10000

Prologue

Summary

• Introduction to Neural Networks and Connectionism
  • Shift from symbolic AI to connectionist approaches in artificial intelligence.
  • Inspiration from biological neural networks and the human brain’s structure.
• Computations with Neurodes and Threshold Logic Units
  • Early models of neurons (neurodes) capable of performing logical operations (AND, OR, NOT).
  • Limitations of simple perceptrons in solving non-linearly separable problems like XOR.
• Multilayer Perceptrons (MLPs) and Feedforward Neural Networks (FNNs)
  • Overcoming perceptron limitations by introducing hidden layers.
  • Structure and information flow in feedforward neural networks.
  • Explanation of forward pass computations in neural networks.
• Activation Functions in Neural Networks
  • Importance of nonlinear activation functions (sigmoid, tanh, ReLU) for enabling learning of complex patterns.
  • Role of activation functions in backpropagation and gradient descent optimization.
  • Universal Approximation Theorem and its implications for neural networks.
• Deep Learning Frameworks
  • Overview of PyTorch and TensorFlow as leading platforms for deep learning.
  • Introduction to Keras as a high-level API for building and training neural networks.
  • Discussion on the suitability of different frameworks for research and industry applications.
• Hands-On Implementation with Keras
  • Loading and exploring the Fashion-MNIST dataset.
  • Building a neural network model using Keras’ Sequential API.
  • Compiling the model with appropriate loss functions and optimizers for multiclass classification.
  • Training the model and visualizing training and validation metrics over epochs.
  • Evaluating model performance on test data and interpreting results.
• Making Predictions and Interpreting Results
  • Using the trained model to make predictions on new data.
  • Visualizing predictions alongside actual images and labels.
  • Understanding the output probabilities and class assignments in the context of the dataset.

3Blue1Brown on Deep Learning

• But what is a Neural Network?
  • youtu.be/aircAruvnKk
  • 19 minutes
• Gradient descent, how neural networks learn?
  • youtu.be/IHZwWFHWa-w
  • 21 minutes
• What is backpropagation really doing?
  • youtu.be/Ilg3gGewQ5U
  • 14 minutes
• Backpropagation calculus
  • youtu.be/Ilg3gGewQ5U

Next lecture

• We will discuss the training algorithm for artificial neural networks.

References

Cybenko, George V. 1989. “Approximation by Superpositions of a Sigmoidal Function.” Mathematics of Control, Signals and Systems 2: 303–14. https://api.semanticscholar.org/CorpusID:3958369.

Géron, Aurélien. 2022. Hands-on Machine Learning with Scikit-Learn, Keras, and TensorFlow. 3rd ed. O’Reilly Media, Inc.

Goodfellow, Ian, Yoshua Bengio, and Aaron Courville. 2016. Deep Learning. Adaptive Computation and Machine Learning. MIT Press. https://dblp.org/rec/books/daglib/0040158.

Hornik, Kurt, Maxwell Stinchcombe, and Halbert White. 1989. “Multilayer Feedforward Networks Are Universal Approximators.” Neural Networks 2 (5): 359–66. https://doi.org/https://doi.org/10.1016/0893-6080(89)90020-8.

Lakoff, George, and Srini Narayanan. 2025. The Neural Mind: How Brains Think. 1st ed. University of Chicago Press. https://press.uchicago.edu/ucp/books/book/chicago/N/bo243406239.html.

LeCun, Yann, Yoshua Bengio, and Geoffrey Hinton. 2015. “Deep Learning.” Nature 521 (7553): 436–44. https://doi.org/10.1038/nature14539.

LeNail, Alexander. 2019. “NN-SVG: Publication-Ready Neural Network Architecture Schematics.” Journal of Open Source Software 4 (33): 747. https://doi.org/10.21105/joss.00747.

McCulloch, Warren S, and Walter Pitts. 1943. “A logical calculus of the ideas immanent in nervous activity.” The Bulletin of Mathematical Biophysics 5 (4): 115–33. https://doi.org/10.1007/bf02478259.

Minsky, Marvin, and Seymour Papert. 1969. Perceptrons: An Introduction to Computational Geometry. MIT Press.

Rosenblatt, F. 1958. “The perceptron: A probabilistic model for information storage and organization in the brain.” Psychological Review 65 (6): 386–408. https://doi.org/10.1037/h0042519.

Russell, Stuart, and Peter Norvig. 2020. Artificial Intelligence: A Modern Approach. 4th ed. Pearson. http://aima.cs.berkeley.edu/.

Marcel Turcotte

[email protected]

School of Electrical Engineering and Computer Science (EECS)

University of Ottawa