Difference between revisions of "Transformer"
(→Context) |
(→Core Idea) |
||
| Line 4: | Line 4: | ||
summary of the 2017 paper “[[Attention]] Is All You Need” by Vaswani et al., which introduced the [[Transformer]] architecture—a paradigm shift in the approach sequence modeling in deep learning: | summary of the 2017 paper “[[Attention]] Is All You Need” by Vaswani et al., which introduced the [[Transformer]] architecture—a paradigm shift in the approach sequence modeling in deep learning: | ||
| − | |||
The paper proposes a novel architecture called the Transformer, which relies entirely on attention mechanisms—no recurrence (RNNs) or convolutions (CNNs). | The paper proposes a novel architecture called the Transformer, which relies entirely on attention mechanisms—no recurrence (RNNs) or convolutions (CNNs). | ||
Revision as of 11:14, 29 July 2025
Meme
A Transformer in AI is a type of neural network architecture that revolutionized how machines process sequential data—like text, audio, or even DNA sequences.
Context
summary of the 2017 paper “Attention Is All You Need” by Vaswani et al., which introduced the Transformer architecture—a paradigm shift in the approach sequence modeling in deep learning:
The paper proposes a novel architecture called the Transformer, which relies entirely on attention mechanisms—no recurrence (RNNs) or convolutions (CNNs).
This design enables parallel processing, faster training, and better handling of long-range dependencies in sequences2.
Key Components
Self-Attention: Each token in a sequence attends to every other token, allowing the model to weigh relationships dynamically.
Multi-Head Attention: Multiple attention layers run in parallel, capturing diverse aspects of token relationships.
Positional Encoding: Since the model lacks recurrence, positional encodings inject order information into the input sequence.
Encoder-Decoder Structure: Both parts use stacked layers of attention and feed-forward networks to process and generate sequences.
⚙️ Technical Advantages Scalability: Fully parallelizable, making it faster to train than RNNs or CNNs.
Performance: Achieved state-of-the-art results on machine translation tasks (e.g., WMT 2014 English-German and English-French).
Generalization: Successfully applied to other NLP tasks like parsing, showing broad applicability.
Impact
Sparked a wave of innovation leading to models like BERT, GPT, T5, and PaLM.
Became one of the most cited papers in AI, reshaping the landscape of natural language processing and beyond.
