Neural Networks in C++: What Happens When Nothing Is Automatic

Type loss.backward() in PyTorch and a few thousand hours of engineering disappear behind one method call. Autograd walks the computation graph, figures out every partial derivative, and hands you gradients you never had to derive. Tensors resize themselves. The BLAS backend picks a matmul kernel tuned for your exact CPU. None of this is visible, which is exactly the point — frameworks exist to make you forget it's happening.

Write the same network in C++ and all of that machinery vanishes. You are the machinery now. Every dimension mismatch, every gradient, every byte of memory is a decision you make explicitly, in advance, with no runtime safety net to catch you if you're wrong. It's a genuinely different way of understanding what a neural network is — less "a model," more "an object with a very unforgiving destructor."

Why Would Anyone Do This on Purpose

Nobody chooses C++ for a neural network because it's convenient. They choose it because:

Performance is deterministic — no interpreter, no garbage collector pausing mid-forward-pass
Memory is fully controllable, down to the byte
The compiler catches type and dimension errors that Python would only discover at runtime, mid-training, three hours in

That combination is why the highest-performance corners of ML still live in C++. The inference engines running behind on-device transcription, embedded vision systems, and llama.cpp-style local LLM runtimes aren't Python underneath — they're hand-tuned C and C++, because at that layer, milliseconds and megabytes are the entire product. Building a small network by hand is essentially an apprenticeship in that world, compressed into a weekend.

Linear Algebra, or: Suddenly You're a Library Author

Every neural network is, underneath the terminology, a pile of matrix multiplications. In Python that's invisible. In C++ you have to decide, explicitly, whether to write your own Matrix class or lean on something like Eigen — and either choice drags in real consequences:

#include <Eigen/Dense>

struct Layer {
    Eigen::MatrixXd W;
    Eigen::VectorXd b;

    Eigen::VectorXd forward(const Eigen::VectorXd& x) const {
        return (W * x + b).unaryExpr([](double v) {
            return v > 0.0 ? v : 0.0;  // ReLU, spelled out
        });
    }
};

That single line, W * x + b, is the entire forward pass of a layer — and it's also where most of the pain lives. Row-major or column-major layout changes which direction through memory is cache-friendly. A careless copy of W on every forward call can quietly turn an O(n²) operation into something far slower, not because the math changed but because the CPU spent its time waiting on RAM instead of computing. None of this shows up as a compiler error. It shows up as "why is this three times slower than it should be," which you then have to profile your way out of.

Forward Propagation: An Equation That Doesn't Forgive Typos

The math is one line:

z = Wx + b
a = σ(z)

The C++ is where every implicit assumption Python made for you becomes a bug you have to prevent yourself. Wrong activation derivative? Silent numerical garbage. Overflowed exponential in a naive sigmoid? nan, propagating through every downstream layer without so much as a warning. There's no framework catching a shape mismatch and throwing a clean RuntimeError: size mismatch — there's undefined behavior, or a segfault, or worse, a program that runs to completion and produces confidently wrong numbers.

Backpropagation: Reimplementing Calculus by Hand

This is the part that actually teaches you something. Autograd frameworks exist specifically so nobody has to hand-derive:

∂L/∂z = ∂L/∂a · σ'(z)
∂L/∂W = ∂L/∂z · xᵀ
∂L/∂b = ∂L/∂z

But writing it yourself means every one of those derivatives has to be correct, by hand, before it ever touches code:

void Layer::backward(const Eigen::VectorXd& dL_dz,
                      const Eigen::VectorXd& x,
                      double learningRate) {
    Eigen::MatrixXd dW = dL_dz * x.transpose();
    W -= learningRate * dW;
    b -= learningRate * dL_dz;
}

Get an index or a transpose wrong and the network doesn't crash — it just fails to learn, or learns something subtly wrong, and you're left debugging calculus instead of code. The standard trick, and arguably the single most useful habit this exercise teaches, is numerical gradient checking: perturb one weight by a tiny ε, measure the change in loss directly, and compare it against what your analytical backprop claims the gradient is. When they disagree, you've found your bug. It's slow, it's tedious, and it's the closest thing C++ has to autograd's safety net.

Memory: Where Good Networks Go to Segfault

Python's garbage collector means you never think about when a tensor gets freed. C++ makes that decision yours, every time:

// The bug that ships at 2am
double* buffer = new double[batchSize * hiddenSize];
train(buffer);
if (earlyStoppingTriggered) return;   // buffer never freed
delete[] buffer;

The fix isn't clever, it's just disciplined:

std::vector<double> buffer(batchSize * hiddenSize);
train(buffer);
if (earlyStoppingTriggered) return;   // destructor runs regardless

RAII and smart pointers (std::unique_ptr, std::vector) turn "did I remember to free this" into "the language handles it," which is most of the benefit of a garbage collector without the runtime overhead of one. Skip that discipline and the failure modes are exactly what you'd expect: leaks that only show up after hours of training, or a segfault that only reproduces on the batch size nobody tested.

Making It Fast: SIMD, Cache Lines, and Other Dark Arts

Here's the counterintuitive part: a hand-written triple-nested loop over your weight matrix will often run slower than a well-optimized Python call into a BLAS library — not because C++ is slow, but because the compiler can't auto-vectorize what it doesn't structurally recognize. Getting real performance out of C++ means:

Laying out memory so the CPU's cache prefetcher can predict your access pattern
Writing loops in a shape the compiler can actually auto-vectorize into SIMD instructions
Avoiding incidental copies of large matrices between function calls
Parallelizing across batches or layers only where the overhead of doing so is actually worth it

This is the humbling lesson of the whole exercise: "low-level" doesn't mean "fast by default." It means fast is now something you have to earn, deliberately, the same way library authors do.

The Tax You Pay for All This Control: Build Systems

None of the above compiles itself. Real projects need CMake to manage headers, sources, and the Eigen dependency; template-heavy numeric code can turn a two-second Python edit-run loop into a forty-second C++ compile, with a wall of template error text if a dimension doesn't line up. It's not glamorous, and it's usually the part people underestimate before they start.

What You Actually Walk Away With

Set against all of that friction, the payoff is real:

Linear algebra stops being an abstraction and becomes something you've laid out in memory yourself
Backpropagation stops being "the framework does it" and becomes a chain rule you've verified by hand
Performance stops being someone else's problem and becomes cache lines, SIMD, and profiler output
Every convenience PyTorch offers becomes visible, specifically because you had to rebuild it

You come out the other side reading framework source code differently — not as magic, but as a set of engineering decisions you now recognize because you've had to make cheaper, buggier versions of the same ones.

Should You Actually Do This?

For learning, yes — build one network, all the way through, by hand. For anything you actually need to ship or scale, no — use the framework; that's what it's for. But it's worth remembering that "hand-rolled C++ neural computation" isn't just a classroom exercise: projects like llama.cpp exist precisely because someone was willing to do, in production, what this article does for practice. The gap between "educational exercise" and "serious infrastructure" here is smaller than it looks.