Deep Learning

How Neural Networks Understand Text, Part 1: From ANN to RNN

See how a feed-forward network represents text, how an RNN processes words in sequence, and why long-term dependencies are difficult to learn.

Sai SrinivasOct 19, 20269 min read

In this article

Neural networks' ability to understand and generate text was a turning point in AI. Today, we see the impact of modern neural networks, especially large language models (LLMs), in our day-to-day lives.

But how did it all begin? What came before the LLM era? How can a bunch of mathematical equations understand our language?

One important step in that journey was the Recurrent Neural Network (RNN).

Before we get to RNNs, let's look at where a traditional feed-forward Artificial Neural Network (ANN) falls short and why we needed something new.

Can a Feed-Forward ANN Understand Text?

Imagine a sentiment classifier that labels a product review as positive or negative.

Review	Sentiment
This product is great	Positive
This product sucks	Negative
Genuinely good product	Positive
Waste of money	Negative

Neural networks work with numbers, so words need a numerical representation before they can be used as input. A simple option is one-hot encoding.

Step 1: Build a vocabulary

Our toy dataset has 10 unique words, treating uppercase and lowercase as the same:

this, product, is, great, sucks, genuinely, good, waste, of, money

Each word gets a vector of length 10, with a 1 in its assigned position and 0 everywhere else:

this     -> [1 0 0 0 0 0 0 0 0 0]
product  -> [0 1 0 0 0 0 0 0 0 0]
is       -> [0 0 1 0 0 0 0 0 0 0]
great    -> [0 0 0 1 0 0 0 0 0 0]

The sentence "This product is great" is now a sequence of four vectors, each with 10 values.

Step 2: Give every review the same input size

A standard fully connected ANN expects a fixed number of input features. The longest review in this small dataset has four words, so we'll use a maximum length of 4.

Shorter reviews need padding. For example, "This product sucks" has three words, so we append an all-zero vector. Then we flatten the four word vectors into one input:

\text{Input size} = 4 \times 10 = 40

For a small network, let's use:

Input layer: 40 features
Hidden layer: 4 neurons
Output layer: 1 neuron, with a sigmoid activation for binary classification

The input-to-hidden layer has:

(40 \times 4) + 4 = \boxed{164}

That's 164 parameters: 160 weights and 4 biases. The output layer adds another 5 parameters, for 169 parameters in total.

Here is the shape of that classifier. To keep the diagram readable, each input node represents one word position (a block of 10 one-hot features). Every block connects to every hidden neuron; in the actual network, each of the 10 features in a block has its own weight.

Feed-forward ANN for sentiment classification

Loading diagram…

flowchart LR
  accTitle: Feed-forward ANN for sentiment classification
  accDescr: Four input blocks, each representing 10 word features, connect to all four hidden neurons. Every hidden neuron connects to the sigmoid output for positive sentiment.
  subgraph input["INPUT LAYER"]
    direction TB
    x1(["Word 1<br/>10 features"])
    x2(["Word 2<br/>10 features"])
    x3(["Word 3<br/>10 features"])
    x4(["Word 4<br/>10 features"])
  end
  subgraph hidden["HIDDEN LAYER"]
    direction TB
    h1(("N1"))
    h2(("N2"))
    h3(("N3"))
    h4(("N4"))
  end
  subgraph output["OUTPUT LAYER"]
    direction TB
    y(("Sigmoid"))
    label(["Positive sentiment<br/>probability"])
  end
  x1 --> h1
  x1 --> h2
  x1 --> h3
  x1 --> h4
  x2 --> h1
  x2 --> h2
  x2 --> h3
  x2 --> h4
  x3 --> h1
  x3 --> h2
  x3 --> h3
  x3 --> h4
  x4 --> h1
  x4 --> h2
  x4 --> h3
  x4 --> h4
  h1 --> y
  h2 --> y
  h3 --> y
  h4 --> y
  y --> label
  style input fill:#111923,stroke:#40566d,color:#c7d7e8
  style hidden fill:#171522,stroke:#514875,color:#ddd5ff
  style output fill:#111c19,stroke:#436956,color:#d4f0e1
  classDef inputNode fill:#1a2938,stroke:#7799bd,color:#edf5ff,stroke-width:1.2px
  classDef hiddenNode fill:#29243b,stroke:#9a8bd0,color:#f0edff,stroke-width:1.5px
  classDef outputNode fill:#1b382d,stroke:#7bc5a2,color:#edfff5,stroke-width:1.6px
  class x1,x2,x3,x4 inputNode
  class h1,h2,h3,h4 hiddenNode
  class y,label outputNode
  linkStyle 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15 stroke:#8190a5,stroke-width:1.2px,opacity:0.58
  linkStyle 16,17,18,19,20 stroke:#79bd9d,stroke-width:2px

Four input blocks, each representing 10 word features, connect to all four hidden neurons. Every hidden neuron connects to the sigmoid output for positive sentiment.

Step 3: See what happens as the input grows

Suppose the maximum review length is 100 words, the one-hot vector size is 100, and the hidden layer still has 32 neurons. The flattened input has 100 × 100 = 10,000 features.

The first dense layer alone needs:

(100 \times 100 \times 32) + 32 = \boxed{320{,}032}

That's 320,032 parameters in the first dense layer. A vocabulary of 100 words is tiny compared with real-world text.

For this flattened, fully connected ANN, two limitations become clear:

Fixed-length inputs can waste work. If a review has only five words, the remaining 95 positions are padding. The network still has a fixed-size input layer for all 100 positions.
The network does not process words one at a time. A flattened input can still tell word order because each position has its own inputs. But the network treats each position separately. It does not reuse the same weights for every word or keep a summary of the words it has already read.

What if we processed one word at a time, kept a summary of earlier words, and reused the same weights for each new word?

That's the idea behind an RNN.

RNN: Processing Text One Word at a Time

A Recurrent Neural Network processes a sequence one timestep at a time. At each timestep, it receives the current word's numerical representation and a hidden state carrying information forward from the previous step.

For "This product is great," we can picture the sequence like this:

RNN unrolled across four words

Loading diagram…

flowchart LR
  accTitle: RNN unrolled across four words
  accDescr: Each word and the previous hidden state enter an identical RNN cell. The first cell receives an initial zero state. The final hidden state feeds a sentiment classifier, and all cells share the same weights.
  h0(("h_0<br/>zeros"))
  subgraph t1["TIME STEP 1"]
    direction TB
    x1(["x_1<br/>This"])
    r1("RNN CELL")
    h1(("h_1"))
    x1 --> r1 --> h1
  end
  subgraph t2["TIME STEP 2"]
    direction TB
    x2(["x_2<br/>product"])
    r2("RNN CELL")
    h2(("h_2"))
    x2 --> r2 --> h2
  end
  subgraph t3["TIME STEP 3"]
    direction TB
    x3(["x_3<br/>is"])
    r3("RNN CELL")
    h3(("h_3"))
    x3 --> r3 --> h3
  end
  subgraph t4["TIME STEP 4"]
    direction TB
    x4(["x_4<br/>great"])
    r4("RNN CELL")
    h4(("h_4"))
    x4 --> r4 --> h4
  end
  h0 -->|"initial state"| r1
  h1 -->|"state"| r2
  h2 -->|"state"| r3
  h3 -->|"state"| r4
  h4 -->|"final state"| pred(["Sentiment<br/>prediction"])
  r1 -. "same weights W_x and W_h" .-> r2
  r2 -.-> r3
  r3 -.-> r4
  style t1 fill:#101720,stroke:#40566d,color:#c7d7e8
  style t2 fill:#101720,stroke:#40566d,color:#c7d7e8
  style t3 fill:#101720,stroke:#40566d,color:#c7d7e8
  style t4 fill:#101720,stroke:#40566d,color:#c7d7e8
  classDef inputNode fill:#1a2938,stroke:#7799bd,color:#edf5ff,stroke-width:1.2px
  classDef cell fill:#29243b,stroke:#9a8bd0,color:#f0edff,stroke-width:1.6px
  classDef state fill:#1b382d,stroke:#7bc5a2,color:#edfff5,stroke-width:1.5px
  classDef prediction fill:#203d31,stroke:#8bd0ac,color:#effff5,stroke-width:1.6px
  class x1,x2,x3,x4 inputNode
  class r1,r2,r3,r4 cell
  class h0,h1,h2,h3,h4 state
  class pred prediction
  linkStyle 0,2,4,6 stroke:#7799bd,stroke-width:1.5px
  linkStyle 1,3,5,7 stroke:#9a8bd0,stroke-width:1.8px
  linkStyle 8,9,10,11,12 stroke:#7bc5a2,stroke-width:1.8px
  linkStyle 13,14,15 stroke:#ba9fea,stroke-width:1.5px,stroke-dasharray:5 4,opacity:0.9

Each word and the previous hidden state enter an identical RNN cell. The first cell receives an initial zero state. The final hidden state feeds a sentiment classifier, and all cells share the same weights.

The same RNN cell is shown at each timestep to make the sequence easier to follow. These are not four separately trained cells: they reuse the same weight matrices, $W_x$ and $W_h$ . The initial state, $h_0$ , is commonly initialized to zeros.

At timestep $t$ , the RNN combines the current input $x_t$ with the previous hidden state $h_{t-1}$ :

h_t = \tanh(W_x x_t + b_x + W_h h_{t-1} + b_h)

The two learned transformations bring the current word and the previous state into the same hidden-state space. The tanh activation then produces the new state, $h_t$ .

For sentiment classification, we can pass the final hidden state through an output layer:

\hat{y} = \sigma(W_o h_T + b_o)

Here, $\hat{y}$ is the predicted probability of positive sentiment. An RNN can process sequences of different lengths; batching sequences of different lengths may still require padding and masking.

This gives the network a way to carry information forward. But a basic RNN can still have trouble learning which information to preserve over many timesteps.

Why RNNs Struggle With Long-Term Dependencies

To see how an RNN unfolds, start with three timesteps. During training, Backpropagation Through Time (BPTT) sends the loss gradient backward through these steps.

For readability, the forward equations below omit biases:

\begin{aligned} h_1 &= \tanh(W_h h_0 + W_x x_1) \\ h_2 &= \tanh(W_h h_1 + W_x x_2) \\ h_3 &= \tanh(W_h h_2 + W_x x_3) \\ \hat{y} &= \sigma(W_o h_3) \end{aligned}

For binary classification, we can use binary cross-entropy:

L = -y\log(\hat{y}) - (1-y)\log(1-\hat{y})

Now extend the example to 10 timesteps. The prediction uses the final state, $\hat{y} = \sigma(W_o h_{10})$ . Since the same $W_h$ is used at every step, its total gradient is the sum of its contributions across all steps. “Direct” means the effect of $W_h$ at one step while holding the previous hidden state fixed. The equation shows six terms; the ellipsis stands for the rest.

\begin{aligned} \frac{\partial L}{\partial W_h} ={}&\underbrace{\frac{\partial L}{\partial h_{10}} \left.\frac{\partial h_{10}}{\partial W_h}\right|_{\text{direct}}}_{t=10} \\ &+ \underbrace{\frac{\partial L}{\partial h_{10}} \frac{\partial h_{10}}{\partial h_9} \left.\frac{\partial h_9}{\partial W_h}\right|_{\text{direct}}}_{t=9} \\ &+ \underbrace{\frac{\partial L}{\partial h_{10}} \frac{\partial h_{10}}{\partial h_9} \frac{\partial h_9}{\partial h_8} \left.\frac{\partial h_8}{\partial W_h}\right|_{\text{direct}}}_{t=8} \\ &+ \underbrace{\frac{\partial L}{\partial h_{10}} \frac{\partial h_{10}}{\partial h_9} \frac{\partial h_9}{\partial h_8} \frac{\partial h_8}{\partial h_7} \left.\frac{\partial h_7}{\partial W_h}\right|_{\text{direct}}}_{t=7} \\ &+ \underbrace{\frac{\partial L}{\partial h_{10}} \frac{\partial h_{10}}{\partial h_9} \frac{\partial h_9}{\partial h_8} \frac{\partial h_8}{\partial h_7} \frac{\partial h_7}{\partial h_6} \left.\frac{\partial h_6}{\partial W_h}\right|_{\text{direct}}}_{t=6} \\ &+ \underbrace{\frac{\partial L}{\partial h_{10}} \frac{\partial h_{10}}{\partial h_9} \frac{\partial h_9}{\partial h_8} \frac{\partial h_8}{\partial h_7} \frac{\partial h_7}{\partial h_6} \frac{\partial h_6}{\partial h_5} \left.\frac{\partial h_5}{\partial W_h}\right|_{\text{direct}}}_{t=5} \\ &+ \cdots \\ &+ \underbrace{\frac{\partial L}{\partial h_{10}} \frac{\partial h_{10}}{\partial h_9} \frac{\partial h_9}{\partial h_8} \cdots \frac{\partial h_2}{\partial h_1} \left.\frac{\partial h_1}{\partial W_h}\right|_{\text{direct}}}_{t=1} \end{aligned}

The $t=10$ contribution passes through no hidden-state transitions before the direct weight effect. The $t=5$ contribution passes through five, and the $t=1$ contribution passes through nine. The same sum applies to the input weights, $W_x$ .

Each hidden-state transition includes the recurrent weights and the derivative of tanh. Multiplying these derivatives across many timesteps can shrink or grow the gradient:

\frac{\partial L}{\partial h_{10}} \frac{\partial h_{10}}{\partial h_9} \frac{\partial h_9}{\partial h_8} \cdots \frac{\partial h_2}{\partial h_1}

If it shrinks repeatedly, early words get a weaker learning signal. If it grows repeatedly, gradients can explode. Ten timesteps do not guarantee either problem, but this equation shows why earlier words pass through more opportunities for the gradient to change.

This can make long-range relationships harder to learn. For example:

The festival was not, despite the rain, road closures, and a delayed start, a failure.

At each timestep, the current word updates the hidden state. Recent words have a more direct influence on the final state, while earlier words must be carried through more timesteps.

This is where Long Short-Term Memory (LSTM) networks come in. Their gates control what information to retain, update, and expose. This can help with long-term dependencies, though vanishing gradients can still occur. In Part 2, we'll look at the LSTM architecture and its equations.

References

CampusX Deep Learning Playlist