Building Functions with ReLU

An interactive companion to Lecture 11

Change the seven parameters of a network with one input, two hidden ReLU units, and one linear output. Start by changing one parameter at a time.

$$f(x)=w_3\varphi(w_1x+b_1)+w_4\varphi(w_2x+b_2)+b_3.$$

Here $\varphi(z)=\max(0,z)$ is ReLU. The coloured curves show the two weighted contributions. The thicker curve is their sum plus the output bias $b_3$.

viewof parameters = {
  const initial = {w1: 1, b1: 1, w3: 1.5, w2: 1, b2: -1, w4: -2, b3: 0};
  const labels = {w1: "w₁", b1: "b₁", w3: "w₃", w2: "w₂", b2: "b₂", w4: "w₄", b3: "b₃"};
  const inputs = Object.fromEntries(
    Object.entries(initial).map(([name, value]) => {
      const input = Inputs.range([-3, 3], {label: labels[name], value, step: 0.1, width: 190});
      input.querySelector('input[type="range"]').setAttribute("aria-label", labels[name]);
      return [name, input];
    })
  );

  const form = Inputs.form(inputs, {
    template: fields => html`<div class="relu-controls">
      <fieldset style="--unit-colour: #c26817">
        <legend>Hidden unit 1</legend>
        <p class="unit-caption">Input weight, bias, output weight</p>
        ${fields.w1}${fields.b1}${fields.w3}
      </fieldset>
      <fieldset style="--unit-colour: #3679b0">
        <legend>Hidden unit 2</legend>
        <p class="unit-caption">Input weight, bias, output weight</p>
        ${fields.w2}${fields.b2}${fields.w4}
      </fieldset>
      <fieldset style="--unit-colour: currentColor">
        <legend>Output</legend>
        <p class="unit-caption">Vertical shift of the complete curve</p>
        ${fields.b3}
      </fieldset>
      <div><button type="button" class="btn btn-outline-secondary btn-sm">Reset parameters</button></div>
    </div>`
  });

  form.querySelector("button").addEventListener("click", () => {
    form.value = initial;
    form.dispatchEvent(new Event("input", {bubbles: true}));
  });

  return form;
}

viewof yExtent = Inputs.select([8, 16, 40, 80], {
  value: 8,
  label: "Vertical range",
  format: extent => `−${extent} to +${extent}`
})

Plot.plot({
  width: Math.min(width, 850),
  height: 410,
  marginLeft: 50,
  marginRight: 20,
  x: {domain: [-3, 3], label: "x", grid: true},
  y: {domain: [-yExtent, yExtent], label: "Output", grid: true},
  color: {
    domain: ["Unit 1 contribution", "Unit 2 contribution", "Network output"],
    range: ["#c26817", "#3679b0", "currentColor"],
    legend: true
  },
  marks: [
    Plot.ruleY([0], {strokeOpacity: 0.35}),
    Plot.ruleX(breakpoints, {
      x: "x", stroke: "series", strokeDasharray: "4,4", strokeOpacity: 0.45,
      clip: true
    }),
    Plot.lineY(samples, {
      x: "x", y: "unit1", stroke: () => "Unit 1 contribution",
      strokeWidth: 2, clip: true
    }),
    Plot.lineY(samples, {
      x: "x", y: "unit2", stroke: () => "Unit 2 contribution",
      strokeWidth: 2, clip: true
    }),
    Plot.lineY(samples, {
      x: "x", y: "output", stroke: () => "Network output",
      strokeWidth: 3.5, clip: true
    })
  ]
})

The axes stay fixed when parameters change. Use the vertical-range selector if a curve extends outside the current view. Dashed vertical lines mark the hidden units’ breakpoints.

html`<div class="relu-breakpoints" aria-live="polite">
  <p><strong>Unit 1:</strong> ${describeBreakpoint(parameters.w1, parameters.b1)}</p>
  <p><strong>Unit 2:</strong> ${describeBreakpoint(parameters.w2, parameters.b2)}</p>
</div>`

From two units to more

Each hidden unit contributes a shifted and scaled ReLU curve. Their sum can change slope wherever either unit changes slope. The output weights can reinforce or oppose the contributions, while $b_3$ moves the entire result vertically.

Imagine adding more hidden units. Where could you place new bends, and how could their output weights help shape the segments between them? Combining more units gives us more building blocks for constructing complex functions, connecting this exploration to the lecture’s discussion of approximation with wider hidden layers.

Try setting $w_3=0$ to remove the first unit’s contribution, or change the sign of $w_4$ to reverse the second unit’s contribution. Use Reset parameters to restore the starting network.

To experiment with more hidden units, try the Neural Network Visualizer. Add units and observe how the network’s regression curve changes. Its hidden units use $\tanh$ activations.

Return to Lecture 11.

How the curves are computed

The browser evaluates the network at points across $[-3,3]$. Breakpoints inside the interval are included explicitly so the plotted line passes through each bend.

relu = z => Math.max(0, z)

breakpoints = [
  {w: parameters.w1, b: parameters.b1, series: "Unit 1 contribution"},
  {w: parameters.w2, b: parameters.b2, series: "Unit 2 contribution"}
].filter(d => d.w !== 0)
  .map(d => ({x: -d.b / d.w, series: d.series}))
  .filter(d => d.x >= -3 && d.x <= 3)

samples = {
  const xs = Array.from({length: 601}, (_, i) => -3 + i / 100);
  xs.push(...breakpoints.map(d => d.x));
  return [...new Set(xs)].sort((a, b) => a - b).map(x => {
    const unit1 = parameters.w3 * relu(parameters.w1 * x + parameters.b1);
    const unit2 = parameters.w4 * relu(parameters.w2 * x + parameters.b2);
    return {x, unit1, unit2, output: unit1 + unit2 + parameters.b3};
  });
}

describeBreakpoint = (w, b) => {
  if (w === 0) return "Constant activation; no breakpoint.";
  const x = -b / w;
  return `x = ${x.toFixed(2)}${x < -3 || x > 3 ? " (outside the plotted interval)" : ""}`;
}