Encoder-Decoder Architectures

Understanding encoder-decoder architectures: principles, mathematics, and applications in sequence-to-sequence tasks.

Encoder-Decoder Architectures: Processing Sequential Data

Section 4.22 - The Sequence-to-Sequence Challenge

Many real-world problems involve transforming one sequence into another sequence, potentially of different lengths. Examples include machine translation (sequence of words to sequence of words), time series forecasting (past values to future values), and music generation (audio features to audio features). This presents unique challenges that cannot be addressed by simple feed-forward architectures.

The fundamental difficulty lies in creating a fixed-size representation of variable-length input that contains sufficient information to generate variable-length output. The encoder-decoder architecture emerged as an elegant solution to this challenge.

Section 4.23 - Mathematical Framework

The encoder-decoder architecture decomposes the sequence transformation problem into two phases. For an input sequence ( x = (x_1, , x_n) ), we want to generate an output sequence ( y = (y_1, , y_m) ), where ( n ) and ( m ) can be different.

The Encoding Phase

The encoder processes the input sequence to create a context vector ( c ):

$$c=f_{\text{enc}}(x_1,\dots ,x_n)$$

where ( f_{} ) is typically a recurrent neural network that produces both outputs and a final state:

$$h_t,s_t=f_{\text{enc}}(x_t,s_{t-1})$$

The final state ( s_n ) serves as the context vector ( c ), capturing the entire input sequence’s information.

The Decoding Phase

The decoder generates the output sequence conditioned on the context vector:

$$y_t=f_{\text{dec}}(c,y_{t-1},s_{t-1}^{\prime })$$

where: - ( s’{t-1} ) is the decoder’s internal state, - ( y{t-1} ) is the previous output, - The initial state ( s’_0 ) is initialized with the context vector ( c ).

Section 4.24 - State Transfer and Conditioning

A crucial aspect of encoder-decoder architectures is how the encoder’s information is transferred to the decoder. The most common approaches are:

1. State Transfer: The encoder’s final state initializes the decoder:

   $$s_0^{\prime }=c=s_n$$

2. Context Conditioning: The context vector is used at each decoding step:

   $$y_t=f_{\text{dec}}(c,y_{t-1},s_{t-1}^{\prime })$$

When using LSTMs, both the cell state and hidden state are transferred:

$$(h_0^{\prime },c_0^{\prime })=(h_n,c_n)$$

This dual state transfer helps maintain both short-term and long-term dependencies.

Section 4.25 - Applications and Variants

Time Series Forecasting

In time series forecasting, the architecture processes known past values to predict future values. The encoder processes the historical sequence, while the decoder generates predictions using: - The encoded historical context, - Any known future information (like calendar features).

The mathematical formulation becomes:

$$y_t=f_{\text{dec}}(c,[y_{t-1},k_t],s_{t-1}^{\prime })$$

where ( k_t ) represents known future features at time ( t ).

Machine Translation

In translation tasks, the encoder processes the source language sentence while the decoder generates the target language translation. The architecture learns to: - Encode semantic meaning from the source language, - Generate grammatically correct sequences in the target language, - Maintain the original message’s intent.

Voice Conversion

For audio processing tasks, the encoder-decoder architecture can transform vocal characteristics while preserving linguistic content. The encoder captures phonetic and prosodic features, while the decoder reconstructs the audio with modified characteristics.

Section 4.26 - Theoretical Properties

The encoder-decoder architecture possesses several important theoretical properties:

1. Information Bottleneck: The context vector ( c ) creates a controlled bottleneck, forcing the model to learn efficient representations of the input sequence.

2. Variable Length Handling: The architecture naturally accommodates input and output sequences of different lengths without requiring padding or truncation.

3. Temporal Abstraction: The encoding phase can learn to abstract temporal patterns at multiple scales, while the decoding phase can generate sequences with different temporal characteristics.

This architectural pattern has become fundamental in sequence processing tasks, particularly when combined with attention mechanisms that allow the decoder to selectively focus on different parts of the input sequence during generation.

Section 4.27 - Exemple Architecture

TKAN Structure