The weighted sum (dot product) has algebraic, statistical and geometrical properties.
In particular the statistics tell you of error correction properties.
Also ReLU is a switch. It is either thrown or not thrown. The dot product of a number of dot products is still a dot product. Put the two concepts together and boom!
You have a lot of insight into how ReLU neural networks work.
https://ai462qqq.blogspot.com/2019/11/artificial-neural-networks.html