Sequence-to-Sequence Models and Attention

An RNN reads data one step at a time. What it takes in and what it gives back depends on the job:

1. One to many: One input, such as an image, produces an output sequence. An image captioner generates one caption as a sequence of words, one word at a time.
2. Many to one: Several inputs lead to one output. A sentiment model reads a review and decides whether it is positive or negative.
3. Many to many: A sequence goes in, and another sequence comes out. This is what happens when a model labels the words in a sentence or translates it.

In part-of-speech tagging, each input word gets one label, so input and output lengths match in a one-to-one alignment. Translation is different: the translated sentence may have a different number of words, and the word order may change.

For example, “I am a very lucky man” has six words in English. Its Telugu translation, “నేను చాలా అదృష్టవంతుడిని”, has three.

A one-to-one RNN labeling setup expects one output for each input step, so it cannot directly represent this length change. An encoder-decoder model handles it in two stages: encode the source, then generate a target sequence until the decoder emits <EOS>.

Sequence-to-Sequence Architecture

In 2014, Ilya Sutskever and colleagues addressed variable-length sequence mapping with an encoder-decoder framework in Sequence to Sequence Learning with Neural Networks.

The model has two parts.

1. Encoder: Reads the input sequence and turns it into a representation.
2. Decoder: Uses that representation to generate the output one token at a time.

The paper used four-layer stacked LSTMs for both the encoder and decoder. It also reversed each source sentence before encoding it. This brought the first source words closer to the start of the decoder's output, making the source and target easier to align during training. The diagrams below show the source in its usual order for readability.

Encoder

Before the sentence reaches the encoder, it is split into tokens. An embedding layer turns each token's ID into a vector of numbers. For notation, call the six embeddings $X_1, \dots, X_6$ and the successive encoder states $h_0, \dots, h_6$.

In the basic sequence-to-sequence model, the encoder discards its intermediate outputs and passes only its final state to the decoder. This summary is the context vector. With an LSTM, the final state includes both the hidden state and the cell state. Bahdanau attention retains the intermediate encoder states so the decoder can attend to them.

Decoder

The decoder starts with a special token, <SOS>, and uses it with the context vector to predict the first target word. We can denote the decoder states by $s_0, \dots, s_4$ and target tokens by $y_0, \dots, y_4$. The diagram shows a Linear layer mapping each hidden state to vocabulary scores, followed by Softmax, which turns those scores into probabilities. The decoder feeds each predicted word into the next step and stops when it produces <EOS>.

Putting the Encoder and Decoder Together

Here, the encoder reads the English sentence and hands its summary to the decoder. The decoder uses it to produce the Telugu translation, one word at a time:

The decoder runs differently during training and inference:

• Training (teacher forcing): At step $t$, the decoder receives the ground-truth token $y_{t-1}$ and learns to predict $y_t$. Using the correct previous token prevents an early mistaken prediction from snowballing through the rest of the training sequence.
• Inference (exposure bias and beam search): At inference time, the decoder must use its own generated tokens. This mismatch with training on ground-truth tokens is called exposure bias. Greedy decoding picks the most likely next token; beam search keeps the top $k$ partial sequences at each step and extends them as generation continues.

Limitations of the Basic Sequence-to-Sequence Model

The fixed-size context vector creates an information bottleneck: a five-word sentence and a fifty-word sentence must both fit into a vector of the same dimension. As the source gets longer, details from early tokens can be lost, and translation quality can drop. BLEU (Bilingual Evaluation Understudy) is an automatic metric for comparing generated translations with references. In the Bahdanau paper's Figure 2, BLEU drops for the fixed-context models on longer sequences.

Attention gives the decoder access to source information at each output step.

Bahdanau Attention

Bahdanau, Cho, and Bengio introduced this approach in Neural Machine Translation by Jointly Learning to Align and Translate. Their encoder uses a bidirectional RNN (BiRNN): one RNN reads the source from left to right, and another reads it from right to left. Their states are concatenated at each position, so each encoder state $h_j$ represents context from both directions.

At each decoding step, attention compares the decoder's previous state with the encoder states. It uses those scores to build a context vector from the source words most relevant to the next prediction.

At step $i$, the context vector $c_i$ is the weighted sum of the encoder states:

Here, $h_j$ is the encoder state at source position $j$, and $\alpha_{ij}$ is the attention weight for that position at step $i$. The alignment network scores each encoder state against the decoder's previous state:

The network adds linear transformations of $s_{i-1}$ and $h_j$ inside the $\tanh$, which is why this is called additive attention. A softmax across source positions converts the scores into weights that sum to one:

The decoder combines its previous state, previous target word, and context vector to compute its new state and predict the next target word:

On the next step, attention calculates a fresh set of weights, letting the decoder focus on different source words as it generates the translation.

In the Bahdanau paper's Figure 2, the RNNsearch-50 model maintains its BLEU score on sentences with 50 words or more. RNNsearch-30 and the fixed-context baselines decline at longer lengths.

Limitations of Bahdanau Attention

At each decoding step, Bahdanau attention scores every pair $(s_{i-1}, h_j)$ with a feed-forward alignment network using $W_a$, $U_a$, $v_a$, and $\tanh$. The source-side projections $U_a h_j$ can be precomputed, but the model still evaluates scores across source and target positions. It also conditions alignment on the previous decoder state $s_{i-1}$, rather than Luong's current state $s_i$.

Luong Attention

In 2015, Minh-Thang Luong, Hieu Pham, and Christopher Manning published Effective Approaches to Attention-based Neural Machine Translation.

Luong attention computes the current decoder state first, then uses it to score the source states:

For content-based attention, the paper describes three scoring functions. It also discusses a separate location-based variant.

• Dot

• General

• Concat

After calculating the context vector $c_i$, Luong attention combines it with the current decoder state to form the attentional vector $\tilde{s}_i$. A final softmax uses this vector to predict the next target word:

Hands-On

This walkthrough builds an LSTM encoder-decoder with Luong dot attention. It assumes X_train and y_train are aligned lists of English and Telugu sentences. Run the blocks in order in one Python session with PyTorch installed. This example keeps source words in their original order. We will tokenize the sentences, batch them with padding, then train the model with teacher forcing.

Building Tokenizer

The tokenizer lowercases English, preserves Telugu text, and maps words to IDs. <SOS> and <EOS> mark sentence boundaries; <PAD> lets us batch sentences of different lengths.

from abc import ABC, abstractmethod
from collections import Counter


class BaseTokenizer(ABC):
    SPECIALS = ["<PAD>", "<UNK>", "<SOS>", "<EOS>"]

    def __init__(self) -> None:
        super().__init__()
        self.stoi = {}
        self.itos = {}

    @abstractmethod
    def _normalize(self, sentence: str) -> str:
        pass

    def _tokenize(self, sentence: str) -> list[str]:
        return self._normalize(sentence).split()

    def _encode(self, sentence: str) -> list[int]:
        ids = [
            self.stoi.get(token, self.stoi["<UNK>"])
            for token in self._tokenize(sentence)
        ]

        return [
            self.stoi["<SOS>"],
            *ids,
            self.stoi["<EOS>"],
        ]

    def fit(self, column: list[str]):
        counter = Counter()

        for sentence in column:
            counter.update(self._tokenize(sentence))

        vocab = self.SPECIALS + list(counter)

        for idx, word in enumerate(vocab):
            self.stoi[word] = idx
            self.itos[idx] = word

        return self

    def transform(self, column: list[str]) -> list[list[int]]:
        return [self._encode(sentence) for sentence in column]

    def __len__(self) -> int:
        return len(self.stoi)


class EnglishTokenizer(BaseTokenizer):
    def _normalize(self, sentence: str) -> str:
        return sentence.lower().strip()


class TeluguTokenizer(BaseTokenizer):
    def _normalize(self, sentence: str) -> str:
        return sentence.strip()

Preparing the Training Dataset

The dataset converts each aligned sentence pair into source and target token IDs. Fit both tokenizers on the training split so words outside the training vocabulary map to <UNK> later.

import torch
from torch.utils.data import Dataset

class TranslationDataset(Dataset):
    def __init__(
        self,
        english_sentences: list[str],
        telugu_sentences: list[str],
        english_tokenizer: EnglishTokenizer,
        telugu_tokenizer: TeluguTokenizer,
    ) -> None:
        super().__init__()

        if len(english_sentences) != len(telugu_sentences):
            raise ValueError("English and Telugu sentence lists must be aligned.")

        self.src = english_tokenizer.transform(english_sentences)
        self.tgt = telugu_tokenizer.transform(telugu_sentences)

    def __len__(self) -> int:
        return len(self.src)

    def __getitem__(self, index: int) -> tuple[torch.Tensor, torch.Tensor]:
        return (
            torch.tensor(self.src[index], dtype=torch.long),
            torch.tensor(self.tgt[index], dtype=torch.long),
        )


# Fit vocabularies on the training split, then encode each aligned pair.
eng_tkr = EnglishTokenizer().fit(X_train)
tel_tkr = TeluguTokenizer().fit(y_train)
train_dataset = TranslationDataset(X_train, y_train, eng_tkr, tel_tkr)

Batching and Padding

The collate function pads source and target sentences separately, using each vocabulary's <PAD> ID.

import torch
from torch.utils.data import DataLoader
from torch.nn.utils.rnn import pad_sequence

def collate_fn(batch: list[tuple[torch.Tensor, torch.Tensor]]):
    source_sequences, target_sequences = zip(*batch)

    source_padded = pad_sequence(
        source_sequences,
        batch_first=True,
        padding_value=eng_tkr.stoi["<PAD>"],
    )
    target_padded = pad_sequence(
        target_sequences,
        batch_first=True,
        padding_value=tel_tkr.stoi["<PAD>"],
    )

    return source_padded, target_padded

train_dataloader = DataLoader(
    dataset=train_dataset,
    batch_size=128,
    collate_fn=collate_fn,
    shuffle=True,
)

Creating the Encoder, Decoder, and Attention Model

The encoder returns its final LSTM state and the hidden output at every source position. The decoder uses dot attention to compare each decoder state with those encoder outputs. A padding mask keeps attention away from padded source positions.

import torch
from torch import nn
from torch.nn.utils.rnn import pack_padded_sequence, pad_packed_sequence


class Encoder(nn.Module):
    def __init__(
        self,
        vocab_size: int,
        embedding_size: int,
        hidden_size: int,
        pad_id: int = 0,
    ) -> None:
        super().__init__()
        self.embedding = nn.Embedding(vocab_size, embedding_size, padding_idx=pad_id)
        self.lstm = nn.LSTM(
            input_size=embedding_size,
            hidden_size=hidden_size,
            batch_first=True,
        )

    def forward(self, source_ids: torch.Tensor, lengths: torch.Tensor):
        embeddings = self.embedding(source_ids)
        packed = pack_padded_sequence(
            embeddings,
            lengths.cpu(),
            batch_first=True,
            enforce_sorted=False,
        )
        packed_outputs, (hidden, cell) = self.lstm(packed)
        outputs, _ = pad_packed_sequence(
            packed_outputs,
            batch_first=True,
            total_length=source_ids.size(1),
        )
        return outputs, hidden, cell


class LuongAttention(nn.Module):
    def forward(
        self,
        decoder_states: torch.Tensor,
        encoder_outputs: torch.Tensor,
        source_mask: torch.Tensor,
    ):
        scores = torch.bmm(decoder_states, encoder_outputs.transpose(1, 2))
        scores = scores.masked_fill(~source_mask.unsqueeze(1), float("-inf"))
        weights = torch.softmax(scores, dim=-1)
        context = torch.bmm(weights, encoder_outputs)
        return context, weights


class Decoder(nn.Module):
    def __init__(
        self,
        vocab_size: int,
        embedding_size: int,
        encoder_hidden_size: int,
        hidden_size: int,
        pad_id: int = 0,
    ) -> None:
        super().__init__()
        if encoder_hidden_size != hidden_size:
            raise ValueError("Dot attention requires equal encoder and decoder hidden sizes.")

        self.embedding = nn.Embedding(vocab_size, embedding_size, padding_idx=pad_id)
        self.attention = LuongAttention()
        self.lstm = nn.LSTM(
            input_size=embedding_size,
            hidden_size=hidden_size,
            batch_first=True,
        )
        self.context_projection = nn.Linear(
            encoder_hidden_size + hidden_size,
            hidden_size,
        )
        self.output_layer = nn.Linear(hidden_size, vocab_size)

    def forward(
        self,
        target_ids: torch.Tensor,
        encoder_outputs: torch.Tensor,
        source_mask: torch.Tensor,
        hidden: torch.Tensor,
        cell: torch.Tensor,
    ):
        embeddings = self.embedding(target_ids)
        decoder_states, (hidden, cell) = self.lstm(embeddings, (hidden, cell))
        context, weights = self.attention(decoder_states, encoder_outputs, source_mask)
        attentional_states = torch.tanh(
            self.context_projection(torch.cat((context, decoder_states), dim=-1))
        )
        logits = self.output_layer(attentional_states)
        return logits, hidden, cell, weights


class Seq2Seq(nn.Module):
    def __init__(
        self,
        encoder: Encoder,
        decoder: Decoder,
        pad_id: int = 0,
    ) -> None:
        super().__init__()
        self.encoder = encoder
        self.decoder = decoder
        self.pad_id = pad_id

    def forward(self, source_ids: torch.Tensor, target_ids: torch.Tensor):
        source_mask = source_ids != self.pad_id
        lengths = source_mask.sum(dim=1)
        encoder_outputs, hidden, cell = self.encoder(source_ids, lengths)
        logits, _, _, attention_weights = self.decoder(
            target_ids,
            encoder_outputs,
            source_mask,
            hidden,
            cell,
        )
        return logits, attention_weights

Initialization

Use each tokenizer's vocabulary size to create the embedding tables and output layer. Dot attention requires matching encoder and decoder hidden sizes. The special-token list puts <PAD> at index zero.

source_vocab_size = len(eng_tkr)
target_vocab_size = len(tel_tkr)
embedding_size = 32
hidden_size = 32
pad_id = eng_tkr.stoi["<PAD>"]

encoder = Encoder(source_vocab_size, embedding_size, hidden_size, pad_id)
decoder = Decoder(
    vocab_size=target_vocab_size,
    embedding_size=embedding_size,
    encoder_hidden_size=hidden_size,
    hidden_size=hidden_size,
    pad_id=tel_tkr.stoi["<PAD>"],
)
model = Seq2Seq(encoder, decoder, pad_id)

Training Loop

The decoder receives the ground-truth previous target tokens during training. The target is shifted by one position so each output is compared with the next word. Padding tokens are excluded from the loss. The model also returns attention weights with shape (batch, target length, source length).

import torch
from torch import nn

epochs = 20
device = "cuda" if torch.cuda.is_available() else "cpu"
model = model.to(device)

optimizer = torch.optim.Adam(
    model.parameters(),
    lr=1e-3,
)
criterion = nn.CrossEntropyLoss(
    ignore_index=tel_tkr.stoi["<PAD>"],
)

epoch_losses = []

for epoch in range(epochs):
    model.train()
    total_loss = 0.0
    total_tokens = 0

    for src, tgt in train_dataloader:
        src = src.to(device)
        tgt = tgt.to(device)
        decoder_input = tgt[:, :-1]
        decoder_target = tgt[:, 1:]
        optimizer.zero_grad()
        logits, _attention_weights = model(src, decoder_input)
        loss = criterion(
            logits.reshape(-1, logits.size(-1)),
            decoder_target.reshape(-1),
        )
        loss.backward()
        nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)
        optimizer.step()

        valid_tokens = (decoder_target != tel_tkr.stoi["<PAD>"]).sum().item()
        total_loss += loss.item() * valid_tokens
        total_tokens += valid_tokens

    avg_loss = total_loss / max(total_tokens, 1)
    epoch_losses.append(avg_loss)
    print(f"Epoch {epoch + 1}: loss = {avg_loss:.4f}")

References

• Sutskever, I., Vinyals, O., and Le, Q. V. (2014). Sequence to Sequence Learning with Neural Networks.
• Bahdanau, D., Cho, K., and Bengio, Y. (2014). Neural Machine Translation by Jointly Learning to Align and Translate.
• Luong, M.-T., Pham, H., and Manning, C. D. (2015). Effective Approaches to Attention-based Neural Machine Translation.
• Papineni, K., Roukos, S., Ward, T., and Zhu, W.-J. (2002). BLEU: a Method for Automatic Evaluation of Machine Translation.
• CampusX. Deep Learning playlist.