Neural Network Weights: A Comprehensive Guide

In machine learning, neural networks use numerical expressions called weights to manipulate input and arrive at the correct output. These numbers are found within the nodes of neural networks and instruct the node about how to interact with the data, including which parts of the data are most important to consider. Neural networks function like human brains, with nodes working like individual neurons and the weights within those nodes working like synapses to emphasize the most important data and disregard data that doesn’t matter. 

Learn how neural networks work and how neural network weights help AI models make complex decisions.

What are neural network weights? 

Neural network weights are numerical values that help each node within a network make decisions by determining which factors are more important than others. A neural network contains layers of nodes, each connecting to nodes in the layer before and the layer after. The nodes manipulate the input data before it moves to the next layer, using weights to put a stronger emphasis on more important connections. 

You may remember how a teacher or professor used a weighted system to calculate high school or college grades. For example, daily homework might represent 15 percent of your final grade, essay assignments 35 percent, and the final exam 50 percent. This allowed your teacher to count the final exam as more important than your essay assignments, but your essays as more important than your daily homework. Your teacher would use a formula to assign weights to each category. If you scored 100 percent on your assignments, 75 percent on your essay assignments, and 85 percent on your final grade, that formula would look like: 

(100*0.15) + (75*0.35) + (85*0.50) = 83.75

Your final score would be 83 or 84 percent, a B or B- grade score, depending on the grading scale. 

Neural networks similarly use weights to make decisions, emphasizing more important factors and taking less consideration of factors with less significance. Neural networks can also adjust these weights using artificial intelligence, increasing their accuracy. To accomplish this, the network will first predict the outcome based on training data, and then compare the final output against its own prediction. By noticing how the final output differs from its prediction, the neural network can adjust its weights and try again, fine-tuning the best weights for its task. Neural network weights are a crucial component of how this AI model learns. 

Understanding weights and biases

While a weight is a numerical value between nodes in separate node layers, a bias is a numerical value that can shift all information after you apply the weights. To understand how these function inside a neural network, reviewing how neural networks work first may be helpful.

Neural networks contain layers of nodes designed to function similarly to neurons in your brain. In neural networks called feedforward networks, the data comes into the input layer and flows through layers of nodes until finally reaching the output layer. Each node connects to nodes in the prior layer and the next layer. When input flows into the node, the node will manipulate the input based on its weights and biases settings before pushing the data through to the next layer, with weights applying to individual connections and biases applying to all the data. 

In other types of neural networks called recurrent networks, the information doesn’t always flow straight through from input to output but may circle back or flow backward. In recurrent neural networks, weights aren’t individualized to each connection but work at the entire layer, similar to a bias in a feedforward network. 

Each node in each layer can make simple decisions. Neural networks, on the other hand, can make complicated decisions because of the number of nodes and layers of nodes they use to arrive at the final output.

Weight initialization

Before training a neural network, you must set starting values for your weights, which you will eventually fine-tune with training. You can choose from several methods of weight initialization, including: 

• Random initialization: A common method is random initialization, where the algorithm selects random numbers depending on the data type and neural network you use. 

• Zero initialization: This method sets all initial weight values to ones or zeros. 

• Constant initialization: This refers to an initialization method in which all the weights are set at the same value to start. 

• LeCun initialization: In this method, you would follow a formula that considers the number of inputs and outputs to create random weights that represent a series of numbers, all with an equal probability of being the starting weight. 

• Xavier initialization: This method is similar to the LeCun method but is used in Keras and takes a square root of the initialization values you would get from a LeCun method. 

• He initialization: A method useful for deep learning, He initialization uses an activation function to assign weights. 

Training neural networks: Adjusting the weights

Once you’ve selected your initialization method, you can instruct the neural network to adjust the weights through backpropagation. In training, the neural network will run an input through its layers of nodes until it reaches an output. It will then calculate the difference between the actual output and the expected output, or what the correct output would have been for that input. Using this calculation, the neural network will change its weights so that the actual output will be closer to or equal to the output, propagating backward through the network. 

Every time you input the neural network, you perform an iteration of training. Working through the entire set of training data in iterations is an epoch. The number of epochs you’ll perform to train your neural network depends on many factors, but getting the calculation right is important.

You’ll need to use enough epochs to train your network, or you risk underfitting or not allowing the AI to fully understand the trends within the training data. If you run too many epochs, you can risk overfitting, when the model is trained so specifically to the training data that it can’t extrapolate to new scenarios. 

Challenges and solutions in weight optimization

In addition to underfitting and overfitting, you may experience challenges such as vanishing and exploding gradients. Explore strategies for overcoming these issues: 

• Underfitting: The model can’t recognize the primary patterns and trends within the training data, so it’s unable to properly process new inputs. This challenge occurs when you need to provide your AI model with additional training, decrease your regularization, or add new features. 

• Overfitting: Your model is too specialized for training data and cannot predict patterns in novel scenarios. This challenge happens when you provide your model with too much additional training, or the training data is overly complex or irrelevant. In this case, you can increase regularization and decrease features. You can also employ validation data, a separate set of training data to validate that you are not under or overfitted. 

• Vanishing gradient: The vanishing gradient problem happens during backpropagation, where the values used to adjust the weights of the nodes diminish or vanish as they work through the algorithm, leading to little to no change or adjustment to the first few layers of the network. You can overcome this challenge by activating functions like Rectified Linear Unit, properly initializing your weights, or applying batch normalization. 

• Exploding gradient: An exploding gradient is the opposite of a vanishing gradient. In this scenario, the figure applied to the weights in backpropagation becomes too large, resulting in erratic behavior and meaningless outputs. You may overcome this challenge as you would address vanishing gradients: implementing batch normalization, using the right activation functions, and properly considering your initial weights. 

Learn more about neural network weights on Coursera

Neural network weights are an important part of how AI models learn and interact with data. If you’d like to learn more about neural networks or train to start working as an AI engineer, courses on Coursera are one way to begin today. Consider Neural Networks and Deep Learning offered as part of the Deep Learning Specialization by DeepLearning.AI. You can also start training for a career with the IBM AI Engineering Professional Certificate.