janmr blog

Neural Networks - Linear Regression

Looking at the basic model for a neural network, it is natural to consider: What is the simplest possible neural network? And is such a network useful for anything?

If we have a single layer (L=1L=1), only one node in the output layer (n1=1n^1=1) and no activation function (g1g^1 is the identity function), then we have a very simple neural network. Obviously, having only one input node (n0=1n^0=1) will be the simplest, but we will initially consider any number of input nodes n01n^0 \geq 1.

Linear regression as a neural network

With this structure and input to the network given by x1,x2,,xn0,x_1, x_2, \ldots, x_{n^0}, we can compute the output zz by

z=j=1n0wjxj+b.z = \sum_{j=1}^{n^0} w_j x_j + b.

(Here we have included a bias bb, as this was also included in the general model.)

So the output of the network is a linear combination of the input values (and the constant 11). Furthermore, as seen from the post on the optimization problem, the error function is a least squares error function.

This means that the simple network described above is equivalent to linear regression (with a least squares error function, as is the most common).

This means that

Feel free to leave any question, correction or comment in this Mastodon thread.