Sequential Data Processing: From MLPs to RNNs

Understanding the challenges of processing financial time series data and why we need recurrent neural networks.

Processing Sequential Data: The Need for Memory

Section 2.1 - The Challenge with Financial Time Series

Limitations of MLPs for Sequential Data

When dealing with financial time series (like price movements, volume, or volatility), the most straightforward approach with an MLP would be to flatten the temporal dimension:

from keras import Sequential, layers

# Example: Using last 10 days of price and volume to predict next day
# Input shape: (batch_size, 10 * 2) where 2 represents price and volume features
model = Sequential([
    layers.Dense(64, activation='relu', input_shape=(20,)),  # 10 timesteps * 2 features
    layers.Dense(32, activation='relu'),
    layers.Dense(1)  # Predict next day's price
])

This approach has fundamental limitations:

1. Loss of Temporal Structure:
   • By flattening the sequence, we lose the natural ordering of time steps
   • All time steps are treated equally in the input layer
   • Network needs to learn temporal relationships from scratch on flattened data
2. Fixed Time Window:
   • Must use same sequence length for training and inference
   • Cannot easily adapt to different historical contexts
   • Need to retrain model for different sequence lengths

Example: Price Prediction Problem

Consider predicting tomorrow’s stock price using 10 days of history:

# Each row is [price_t, volume_t, price_t-1, volume_t-1, ..., price_t-9, volume_t-9]
X = np.array([
    [100, 1000, 98, 1100, 97, 900, ...],  # Recent to old
    [101, 1200, 100, 1000, 98, 1100, ...],
    ...
])

The MLP sees this as just a vector of numbers, losing the inherent sequential nature of the data.

Section 2.2 - The Need for Temporal Processing

Step-by-Step Processing

A more natural approach is to process the sequence one time step at a time:

# Conceptual representation of sequential processing
def process_financial_sequence(prices, volumes):
    state = initialize_state()
    for price, volume in zip(prices, volumes):
        # Update state with new market information
        state = update_state(state, price, volume)
    return state

This raises important questions for financial data: 1. How do we track market trends over different time scales? 2. How do we maintain relevant historical context? 3. How do we learn what patterns are predictive?

The Memory Challenge in Financial Data

Financial Data Characteristics

• Markets exhibit both short-term and long-term patterns
• Recent data typically matters more (but not always)
• Need to detect regime changes and trend reversals
• Must handle varying levels of volatility

Requirements for our solution: 1. Adaptive Memory: - Track relevant market conditions - Detect pattern changes - Handle multiple time scales

2. Learnable Pattern Recognition:
   • Automatically identify important market signals
   • Adapt to changing market conditions
   • Balance recent and historical information

Section 2.3 - Introducing Recurrence

The Basic Concept

# Simplified conceptual RNN for financial data
class MarketRNN:
    def step(self, market_data_t, h_t_prev):
        # Combine current market data with previous state
        h_t = tanh(W_h @ h_t_prev + W_x @ market_data_t + b)
        return h_t

    def process_sequence(self, price_history, volume_history):
        h_t = initialize_hidden_state()
        for price_t, volume_t in zip(price_history, volume_history):
            market_data_t = np.array([price_t, volume_t])
            h_t = self.step(market_data_t, h_t)
        return h_t

Key advantages for financial data: 1. State Maintenance: - Tracks market conditions over time - Updates state with each new data point - Maintains relevant historical context

2. Sequential Processing:
   • Processes data in chronological order
   • Can learn temporal dependencies
   • Natural handling of time series
3. Efficient Parameters:
   • Same weights process each time step
   • Learns general temporal patterns
   • Can generalize across different market conditions

Tip

This basic recurrent structure will lead us to more sophisticated architectures like LSTM and GRU, which are particularly effective for financial time series due to their ability to handle long-term dependencies.