Assignment : Recurrent Neural Networks

Table of Contents
1. Architecture and Core Concepts
2. Training RNNs
3. Challenges in Training RNNs
4. Activation Functions in RNNs
5. Variants and Improvements
6. Applications of RNNs
7. Implementation Considerations
8. Common Mistakes and Traps
9. Performance Metrics and Evaluation
View more

Recurrent Neural Networks (RNNs) are a specialized class of neural networks designed to process sequential data by maintaining hidden states that capture temporal dependencies. Unlike feedforward networks, RNNs have loops that allow information to persist across time steps, making them ideal for tasks involving sequences such as time series prediction, language modeling, and speech recognition. Understanding RNNs is crucial for mastering sequence modeling and natural language processing tasks.

1. Architecture and Core Concepts

1.1 Basic RNN Structure

• Recurrent Connection: RNNs have a feedback loop where the hidden state at time step t depends on both the current input x_t and the previous hidden state h_t-1.
• Parameter Sharing: The same weight matrices (W_xh, W_hh, W_hy) are used across all time steps, enabling the network to generalize across different positions in the sequence.
• Hidden State: Acts as the network's memory, capturing information from previous time steps. It is updated at each time step using a recurrent formula.
• Unfolding in Time: RNNs can be visualized as a chain of repeated modules when unrolled across time steps, where each module processes one element of the sequence.

1.2 Mathematical Formulation

The core computations in a vanilla RNN are defined by the following equations:

• Hidden State Update: h_t = tanh(W_hh · h_t-1 + W_xh · x_t + b_h)
  • h_t: hidden state at time t
  • h_t-1: previous hidden state
  • x_t: input at time t
  • W_hh: weight matrix for hidden-to-hidden connections
  • W_xh: weight matrix for input-to-hidden connections
  • b_h: bias term for hidden state
  • tanh: hyperbolic tangent activation function
• Output Computation: y_t = W_hy · h_t + b_y
  • y_t: output at time t
  • W_hy: weight matrix for hidden-to-output connections
  • b_y: bias term for output
• Initial Hidden State: h₀ is typically initialized to zeros or learned as a parameter.

1.3 Types of RNN Architectures

• One-to-One: Standard neural network with no sequence (included for completeness in taxonomy).
• One-to-Many: Single input produces a sequence output (e.g., image captioning where an image generates a sentence).
• Many-to-One: Sequence input produces a single output (e.g., sentiment analysis where a sentence is classified into positive/negative).
• Many-to-Many (Synced): Input and output sequences have the same length (e.g., video classification at each frame).
• Many-to-Many (Encoder-Decoder): Input and output sequences have different lengths (e.g., machine translation from English to French).

2. Training RNNs

2.1 Backpropagation Through Time (BPTT)

• Concept: BPTT is the standard algorithm for training RNNs. It unfolds the network across time steps and applies backpropagation to compute gradients.
• Forward Pass: Compute hidden states and outputs sequentially from t = 1 to t = T (sequence length).
• Backward Pass: Calculate gradients by backpropagating errors from the final time step back to the first, accumulating gradients at each step.
• Gradient Computation: Gradients flow backward through time, requiring the chain rule to be applied across all time steps.

2.2 Truncated Backpropagation Through Time

• Purpose: Used when sequences are very long to reduce computational cost and memory requirements.
• Method: Divide the sequence into fixed-length chunks (e.g., k time steps) and perform BPTT only within each chunk.
• Forward Pass Continuity: Hidden states are carried forward across chunks, but gradients are not backpropagated beyond the chunk boundary.
• Trade-off: Reduces memory usage but limits the network's ability to capture long-term dependencies.

2.3 Loss Functions

• Sequence Classification: Cross-entropy loss applied to the final output (many-to-one architecture).
• Sequence Generation: Sum of cross-entropy losses at each time step for predicting the next token (many-to-many architecture).
• Total Loss: L = Σ_t=1^T L_t, where L_t is the loss at time step t.

3. Challenges in Training RNNs

3.1 Vanishing Gradient Problem

• Cause: During BPTT, gradients are multiplied repeatedly by the weight matrix W_hh as they propagate backward through time.
• Effect: If the largest eigenvalue of W_hh is less than 1, gradients shrink exponentially, approaching zero for early time steps.
• Consequence: The network fails to learn long-term dependencies because gradients from distant time steps become too small to update weights effectively.
• Typical Range: Vanishing occurs when gradients become smaller than 10^-6 to 10^-8.

3.2 Exploding Gradient Problem

• Cause: If the largest eigenvalue of W_hh is greater than 1, gradients grow exponentially during backpropagation.
• Effect: Gradients become extremely large (e.g., exceeding 10¹⁰), causing numerical instability and NaN values in parameters.
• Solution - Gradient Clipping: Limit the norm of gradients to a maximum threshold value (e.g., clip if ||g|| > threshold, rescale to g = g × threshold/||g||).
• Threshold Values: Commonly set between 1 and 10 depending on the problem.

3.3 Difficulty in Capturing Long-Term Dependencies

• Memory Limitation: Vanilla RNNs struggle to remember information from more than 5-10 time steps in the past due to vanishing gradients.
• Information Decay: The influence of early inputs diminishes exponentially as the sequence length increases.
• Practical Implication: Tasks requiring context from 50+ time steps ago (e.g., long documents) perform poorly with basic RNNs.

4. Activation Functions in RNNs

4.1 Hyperbolic Tangent (tanh)

• Formula: tanh(x) = (e^x - e^-x)/(e^x + e^-x)
• Range: Output values lie between -1 and +1, providing zero-centered activations.
• Advantage: Zero-centering helps with gradient flow compared to sigmoid function.
• Usage: Most common activation for hidden state computation in vanilla RNNs.

4.2 ReLU (Rectified Linear Unit)

• Formula: ReLU(x) = max(0, x)
• Advantage: Helps mitigate vanishing gradients since gradient is 1 for positive inputs.
• Limitation: Can cause exploding activations in RNNs if not carefully initialized.
• Usage: Sometimes used in specialized RNN variants but less common than tanh.

4.3 Sigmoid Function

• Formula: σ(x) = 1/(1 + e^-x)
• Range: Output values between 0 and 1.
• Usage in RNNs: Primarily used in gating mechanisms (LSTM and GRU) rather than for hidden state activation in vanilla RNNs.
• Limitation: Suffers from vanishing gradient problem more severely than tanh due to gradients saturating at both extremes.

5. Variants and Improvements

5.1 Deep RNNs

• Structure: Stack multiple RNN layers vertically where the output of one layer becomes the input to the next layer.
• Computation: h_t^(l) = tanh(W_hh^(l) · h_t-1^(l) + W_xh^(l) · h_t^(l-1) + b_h^(l)), where l denotes layer number.
• Advantage: Increases representational capacity by learning hierarchical features at different layers.
• Common Depth: Typically 2-4 layers for RNNs (deeper than feedforward networks are rare due to training difficulties).

5.2 Bidirectional RNNs

• Architecture: Contains two separate RNN layers - one processes the sequence forward (left to right), the other processes it backward (right to left).
• Hidden State: h_t = [h_t^forward; h_t^backward] (concatenation of forward and backward hidden states).
• Advantage: Can access both past and future context at each time step, useful when entire sequence is available.
• Applications: Named entity recognition, part-of-speech tagging, speech recognition where future context helps prediction.
• Limitation: Cannot be used for real-time prediction tasks where future inputs are not yet available.

5.3 LSTM and GRU (Brief Overview)

• Purpose: Advanced RNN architectures designed to solve the vanishing gradient problem and capture long-term dependencies.
• LSTM (Long Short-Term Memory): Uses memory cells and three gates (forget, input, output) to control information flow.
• GRU (Gated Recurrent Unit): Simplified version with two gates (reset, update), computationally more efficient than LSTM.
• Key Difference from Vanilla RNN: Gating mechanisms allow selective retention and forgetting of information across long sequences.

6. Applications of RNNs

6.1 Language Modeling

• Task: Predict the next word in a sequence given previous words.
• Formulation: P(w_t | w₁, w₂, ..., w_t-1) - probability of word at position t given all previous words.
• Training: Use cross-entropy loss between predicted probability distribution and actual next word.
• Evaluation Metric: Perplexity = 2^{average cross-entropy loss}, lower perplexity indicates better model.

6.2 Machine Translation

• Encoder-Decoder Architecture: Encoder RNN processes source language sequence into a fixed-length context vector; decoder RNN generates target language sequence from this vector.
• Context Vector: Final hidden state of encoder, capturing the meaning of entire source sentence.
• Limitation: Fixed-length context vector creates an information bottleneck for long sentences (addressed by attention mechanisms).

6.3 Speech Recognition

• Input: Sequence of audio features (e.g., MFCCs - Mel-Frequency Cepstral Coefficients) extracted from speech signal.
• Output: Sequence of phonemes or directly transcribed text.
• Architecture: Bidirectional RNNs are commonly used since entire audio is available before transcription.

6.4 Time Series Prediction

• Task: Forecast future values based on historical observations (e.g., stock prices, weather, sensor data).
• Approach: Many-to-one RNN for single-step prediction or many-to-many for multi-step forecasting.
• Challenge: Capturing both short-term patterns and long-term trends in the data.

7. Implementation Considerations

7.1 Input Representation

• One-Hot Encoding: Represent each word/token as a vector with vocabulary size dimensions, with a single 1 and rest 0s.
• Word Embeddings: Dense, low-dimensional vector representations (e.g., 50-300 dimensions) learned during training or pre-trained (Word2Vec, GloVe).
• Advantage of Embeddings: Capture semantic relationships and reduce dimensionality compared to one-hot vectors.

7.2 Sequence Padding and Masking

• Padding: Add special tokens (e.g., <PAD>) to make all sequences in a batch the same length for efficient computation.
• Masking: Use a mask to indicate which positions are actual data vs padding, ensuring loss is not computed on padded positions.
• Positioning: Padding can be added at the beginning (pre-padding) or end (post-padding) of sequences.

7.3 Weight Initialization

• Xavier/Glorot Initialization: Initialize weights with variance scaled by fan-in and fan-out to maintain gradient magnitude.
• Orthogonal Initialization: Initialize recurrent weight matrix W_hh as an orthogonal matrix to help preserve gradient norm during backpropagation.
• Identity Initialization: Initialize W_hh close to identity matrix with ReLU activation to encourage gradient flow.

7.4 Regularization Techniques

• Dropout: Apply dropout to non-recurrent connections (input-to-hidden, hidden-to-output) but not to recurrent connections to avoid disrupting temporal flow.
• Recurrent Dropout: Specialized dropout that uses the same dropout mask at every time step rather than different masks.
• L2 Regularization: Add penalty term λ||W||² to loss function to prevent weights from growing too large.
• Early Stopping: Monitor validation loss and stop training when it stops improving to prevent overfitting.

8. Common Mistakes and Traps

8.1 Trap: Hidden State Management

• Mistake: Not resetting hidden state between independent sequences in a batch, causing information leakage.
• Correction: Initialize hidden state to zero at the start of each new sequence or document.
• Exception: For stateful RNNs processing continuous streams, carry forward hidden states across batches intentionally.

8.2 Trap: Gradient Clipping Threshold

• Mistake: Setting gradient clipping threshold too high (ineffective) or too low (limiting learning).
• Counter-intuitive Fact: Even with gradient clipping, exploding gradients can still cause training instability if the threshold is not tuned properly.
• Best Practice: Monitor gradient norms during training and adjust threshold based on observed values.

8.3 Trap: Sequence Length vs Memory

• Confusion: Assuming vanilla RNNs can handle arbitrarily long sequences simply because they have recurrent connections.
• Reality: Effective memory span of vanilla RNNs is typically limited to 5-10 time steps due to vanishing gradients.
• Solution: Use LSTM/GRU for sequences requiring longer memory, or apply attention mechanisms.

8.4 Trap: Bidirectional RNNs for Prediction

• Mistake: Using bidirectional RNNs for real-time sequential prediction tasks where future inputs are unavailable.
• Explanation: Bidirectional RNNs require the entire sequence to be available before processing, making them unsuitable for streaming applications.
• Correct Usage: Use standard (unidirectional) RNNs for online prediction tasks like next-word prediction in text generation.

9. Performance Metrics and Evaluation

9.1 Language Tasks

• Perplexity: Measures how well the probability distribution predicted by the model matches actual distribution; lower is better.
• BLEU Score: For machine translation, measures n-gram overlap between generated and reference translations (0 to 1 scale, higher is better).
• Accuracy: For classification tasks like sentiment analysis, percentage of correctly classified sequences.

9.2 Time Series Tasks

• Mean Absolute Error (MAE): Average absolute difference between predicted and actual values.
• Root Mean Square Error (RMSE): Square root of average squared differences, penalizes large errors more heavily.
• Mean Absolute Percentage Error (MAPE): Percentage-based error metric useful for comparing across different scales.

9.3 Sequence Labeling Tasks

• Token-Level Accuracy: Percentage of correctly predicted labels across all tokens in all sequences.
• F1 Score: Harmonic mean of precision and recall, useful when classes are imbalanced (e.g., named entity recognition).
• Sequence Accuracy: Percentage of sequences where every token is correctly labeled (stricter metric).

Recurrent Neural Networks form the foundation for sequence modeling in deep learning, introducing the concept of temporal dependencies through recurrent connections. While vanilla RNNs face challenges like vanishing gradients and limited memory span, understanding their architecture and training dynamics is essential before moving to advanced variants like LSTM and GRU. Key exam points include the mathematical formulation of RNN computations, BPTT algorithm mechanics, gradient problems and their solutions, architectural variations (deep and bidirectional), and appropriate application scenarios. Mastery of these fundamentals enables effective sequence modeling and forms the basis for modern natural language processing systems.